This is on Core using TrueNAS 13.
My 8TB pool filled up about 1T during the nighttime and I expect it filled up completely. This has consequences, I know, but I’m trying to save the data.
The errors I get at the time of importing the pool:
Syncing all disks complete!
Alarm clock
Starting file system checks:
Mounting local filesystems:.
Beginning pools import
Importing Main Pool
vdev.c: 161:vdev_dbgmsg(): disk vdev '/dev/ada1': best uberblock found for spa $import. txg 44704638
spa_misc.c: 419:5pa_load_note(): spa_load($import, config untrusted): using uberblock with txg=44704638
spa.c: 8392:spa_async_request(): spa=$import async request task=2048 spa_misc.c:419:5pa_load_note(): spa_load($import, config trusted): LOADED spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): UNLOADING spa.c:6110:spa_import(): spa_import: importing Main Pool
spa_misc.c: 419:5pa_load_note(): spa_load (Main Pool, config trusted): LOADING vdev.c: 161:vdev_dbgmsg(): disk vdev: /dev/ada0': best uberblock found for spa Main Pool. txg 44704638
spa_misc.c: 419:5pa_load_note(): spa_load (Main Pool, config untrusted): using uberblock with txg=44704638
That last bit keeps repeating a few times with different txg values.
After that it runs out of memory it seems:
pid 322 (python3.9), jid 0, uid 0, was killed: failed to reclaim memory
pid 486 (python3.9), jid 0, uid 0, was killed: failed to reclaim memory
pid 485 (python3.9), jid 0, uid 0, was killed: failed to reclaim memory
pid 468 (trace), jid 0, uid 0, was killed: failed to reclaim memory
pid 347 (python3.9), jid 0, uid 0, was killed: failed to reclaim memory
pid 352 (python3.9), jid 0, uid 0, was killed: failed to reclaim memory
pid 467 (python3.9), jid 0, uid 0, was killed: failed to reclaim memory
And it crashes.
What I’ve tried:
- Boot with Single User mode enabled: works, but importing pool fails similarly. I can import with
-o readonly=onflag but that doesn’t allow me to fix an issue. I also don’t have 8+TB of storage laying around to copy it to. zdb -e -bcsv "Main Pool"- Took 3 days, didn’t fix anythingzfs list -t snapshot -r "Main Pool"→ remove snapshots (it only had a few for system, few MB total)zfs create -V 32G -o org.freebsd:swap=on -o checksum=off -o compression=off -o dedup=off -o sync=disabled -o primarycache=none boot-pool/swapCreate a swap file on the boot disk (128GB SSD) but it doesn’t seem to use it.
I’ve tried plenty of other things that didn’t work, like going to a previous working txg with zpool import -T xxxxxxxx but that gives an error that there is metadata corruption. Importing with -F (rewind) -FX (rewind a lot further) but both did nothing.
Is there anything left to try ?
