These appear to be netdata monitoring system cache/core dumps — not user data. This is potentially very good news if my actual data files are intact.
System hardware specs (for your reference):
CPU: Intel Core i7-8700 @ 3.20GHz
RAM: 64GB (4x16GB, appears non-ECC based on dmidecode output)
Storage controllers: Mixed — 3 of the 4 RAIDZ1 disks connected via onboard Intel SATA/AHCI, but 1 disk (sda, 8TB) connected via Broadcom LSI SAS2008 HBA over SAS transport
TrueNAS: Community Edition 25.10.5 Goldeye (custom build, not official appliance)
What would you recommend as the next step? And thank you again for your thorough guidance — I really appreciate your expertise and patience!
Is there a reason all 4 can’t be connected to the motherboard directly?
Doesn’t hurt to run several memtest passes to rule out bad RAM. Something caused corruption. If not a software or ZFS bug, then I wouldn’t rule out RAM.
If your RAM passes several memtests and there’s no safe way to connect all the drives directly to the motherboard, then I’m afraid you might have to backup and rebuild the pool. This is possible with a readonly pool. You’ll need to use -N and a custom root mount path with full permissions.
Not yet though. First, try the simple steps. At least you know you can import the pool as readonly.
EDIT: Someone more versed in lower level ZFS might know a way to safely import your pool normally or use a rollback.
Thank you for your guidance! I’ve completed the steps you recommended.
**Memtest86+ Results:** I ran Memtest86+ v8.10 overnight and let it run for over 19 hours. Result: **8 complete passes, zero errors**. The RAM (4x16GB DDR4-3200 mix Micron/Kingston) is therefore ruled out as the cause.
**About the disk configuration:** You asked why I didn’t have all 4 disks directly on the motherboard. Here’s the explanation:
The ASUS Prime Z370-P motherboard has only **3 SATA ports** (plus 1 M.2 NVMe for boot)
I initially created the RAIDZ1 pool with 3 Seagate IronWolf 8TB disks
When I wanted to expand later, no more SATA ports were available, so I added a Broadcom/LSI SAS2008 HBA card with the 4th disk — that’s why only 1 of the 4 disks goes through the SAS controller, the other 3 stay on AHCI
Mixing transports (AHCI plus SAS) in the same RAIDZ vdev probably isn’t ideal and might be related to the issue.
Mixed SAS/SATA transport shouldn’t be an issue here, even less so because you’re using two separate controllers for it.
The source of the fault is probably going to be buried in the ZFS debug log during an import (a non-readonly one) so I would go back to the zpool import -f -R /mnt 7835538760692273829 pool1 command, and then depending on Linux vs FreeBSD:
Linux: tail /proc/spl/kstat/zfs/dbgmsg
FreeBSD: echo zfs_dbgmsg | mdb -k
Look at the last 20-30 lines once it stalls. Let’s see if it’s hanging up on the same spot or if it’s just furiously swapping metaslabs in and out.
I ran the debug log capture as you suggested. The zpool import is indeed stuck in a loop furiously swapping metaslabs in and out on vdev_id 0 — loading, then immediately unloading (346-347ms each time), then repeating with different metaslab IDs (903, 15, 848, 874, 878, 885, 887…). The old_weight and new_weight values keep changing.
This looks exactly like what you predicted — not hanging at one spot, but endlessly cycling through metaslab operations on the expanded vdev.
If you want to discuss why and when one should run memory tests, if “FreeBSD deals with drive errors better than Linux” or “why TN/Linux fails when other OSes don’t.”, I ask you to post topics concerning those areas in the General Discussion section. Not in-line in a thread where community members are helping another with a specific issue.
Disrupting this thread is the opposite of helpful.
I have no special powers in this forum, so all I can do is ask you to please cease the type of posts you made in this thread over the last couple of days.
There’s a stack of unflushed_allocs and unflushed_frees in all of those metaslab juggles.
Are you by any chance using deduplication, or heavily leveraging block clone (lots of copy-in-place?) that might make it have to handle a lot of behind-the-scenes cleanup?
I do use Jellyfin (media server) which continuously reads video files from the pool. However, I don’t use deduplication, block clone, or any complex backup software.
Just before the last shutdown (when the bug occurred), I deleted a large batch of files (movies) via SMB share. I didn’t think about it at the time, but that could have generated a lot of pending “frees” that weren’t flushed before the shutdown.
Could that explain the unflushed_allocs and unflushed_frees stack?
A big batch of deletes would account for your unflushed_frees but not the unflushed_allocs - did you move anything between datasets for reorganization purposes?
