Pool import causes panic then reboot and no access to data if read-only

TrueNAS General
,
1

6700k / Z170 mobo
1x 250G Nvme
1x 1T SATA3
6x 14T WD Red/Seagate Iron wolf
Gigabyte GC-RLE086-RH LSI 1068E HBA card
64gb non-ecc memory

Long story short is that I had a bad memory stick that caused data corruption, which has since been removed after locating it with memtest, then a couple weeks later I had a power cut which then caused the server to continually boot loop. When plugged into a display, this looks to be because of an issue when it trys to import the zfs pool.

(apologies for the literal screen shot, I cant work out how to get the panic log out as text).

(imgur .com/a/WTOy6ZT)

I have reinstalled the OS as a clean install, and if I try to import the pool using the GUI or the CLI, it’ll repeat the same behaviour.

If I use the CLI to import the pool as a read only pool:

sudo zpool import -F -f -o readonly=on GNAS

It says I dont have access to the underlying folders/data, I cant see how much space, no chown/chmod etc.

My plan was to get the important data (folder of photos) off the pool and just create a new one. I used a 1t sata drive and created a striped pool (photos) on it just for somewhere to put the data, whilst I deleted and rebuilt the large pool, however being unable to access the read-ony pool, I cant copy the data.

zpool list:

% zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
GNAS       76.2T      0  76.2T        -         -     0%     0%  1.00x    ONLINE  -
boot-pool   206G  1.30G   205G        -         -     0%     0%  1.00x    ONLINE  -
photos      928G  11.1M   928G        -         -     0%     0%  1.00x    ONLINE  /mnt

status of the big pool:

% zpool status GNAS
  pool: GNAS
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: <redacted for post>
  scan: scrub in progress since Wed Jun 19 02:13:16 2024
        0B scanned at 0B/s, 0B issued at 0B/s, 43.4T total
        0B repaired, 0.00% done, no estimated completion time
config:

        NAME                                            STATE     READ WRITE CKSUM
        GNAS                                            ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/a98ee38b-b161-11ed-a304-0015178a3995  ONLINE       0     0     0
            gptid/4c31eed1-9fca-11ed-ba8a-0015178a3995  ONLINE       0     0     0
            gptid/69d2469f-0a03-11ed-bbed-0015178a3995  ONLINE       0     0     0
            gptid/c8f635d9-9fc9-11ed-ba8a-0015178a3995  ONLINE       0     0     0
            gptid/83355a92-7c2e-11ee-b8a2-0015178a3995  ONLINE       0     0     0
            gptid/7e41ea2f-0a03-11ed-bbed-0015178a3995  ONLINE       0     0     0

errors: 315 data errors, use '-v' for a list

If I try to list the data errors (as sudo), it returns permission denied:

errors: List of errors unavailable: permission denied

And nothing has any permission once within the mounted pool:

% ls -lth
ls: 3D Models: Permission denied
ls: Adam: Permission denied
ls: Alison: Permission denied
ls: Drivers ISOs Installers etc: Permission denied
ls: Eleanor: Permission denied
ls: Houses: Permission denied
ls: Josephine: Permission denied
ls: Lawrie: Permission denied
ls: lazad: Permission denied
ls: Lottie: Permission denied
ls: Media: Permission denied
ls: Photos: Permission denied
ls: Scans: Permission denied
ls: Wedding: Permission denied
total 0

The key is to get the Photos and Wedding Folders off the big pool (wife factor), but if I could mount the whole thing, thatd be better!

My hope is it can be sorted within ZFS, else ill have to hope something like Klennet sorts it.

Happy to provide any outputs anyone needs.

2

According to this, the pool has nothing on it?

3

But it definitely does! I dont know why its saying access denied, and then not showing contents.

4

You can see in the last scrub details, it says about it containing 43T of data!

5

That’s very disconcerting.

@HoneyBadger, is there a scenario where importing a pool as “readonly” with the -F and -f flags would report a (supposed) completely empty pool?

I highly doubt that a previous TXG was during a point in time where the pool was completely empty, unless it’s a brand new pool that just started being used.

@Lazadude, any reason why you invoked -F and -f, instead of only using “readonly”?

6

Joys of copying commands from Reddit, this trail (as I cant put links in here, working around with double spaces…)

reddit. com/r/truenas/comments/15nn5hr/kernel_panic_when_importing_pools/

7

Pool itself is 7+ years old. Been through various upgrade cycles of FreeNAS/TrueNAS, first time its been an issue, and I can’t work out why.

8

Also, cant set it to read/write either, as I cant set anything against the pool:

% zfs set readonly=off GNAS
cannot set property for ‘GNAS’: pool is read-only
% zpool set readonly=off GNAS
cannot set property for ‘GNAS’: property ‘readonly’ can only be set at import time

9

This is because the “readonly” pool property can only be set during import.

I wouldn’t go messing around with anything, until someone with more familiarity can chime in.

10

Yeah, I’ll stop for a bit. Fun of fiddling…

11

I am not an expert, so ideally someone else will give you good advice. But as I have been going through something similar and replies are slow on this forum, I wanted to mention this GitHub issue thread that has some helpful replies: thread.

The main useful thing I got from that thread was setting zfs_recover and zil_replay_disable which allowed me to import my corrupted pool without -f nor readonly and it prevented a kernel panic.

These flags can be set during runtime with echo:

echo 1 > /sys/module/zfs/parameters/zil_replay_disable
echo 1 > /sys/module/zfs/parameters/zfs_recover

Or before boot by editing GRUB options to add:

zfs.zil_replay_disable=1
zfs.zfs_recovery=1

However, since your issue seems a bit different, these zfs parameters may not be helpful, and to restate: I am just following the directions of others same as you.

12

Yeah, I tried a variety of ZFS setting changes and it still forces a cpu panic and reboot before posting here.

Going to install windows and see what I can get on Klennet, since I need the NAS up.

Really debating throwing out the mobo etc and getting something with ECC as that would have stopped this headache!