-fFn reports no errors in shell.
however I can see some stuff in the log file mentioned previously. Can someone confirm if this means it failed even the -F import? I think that’s what the last few lines mean but not an expert.
Let me know.
Log2.txt (142.4 KB)
No error is supposed to mean (potential) success, but the log seems to imply otherwise. In ‘dry-run’ (-n) nothing will happen anyway.
This is where @HoneyBadger or @Arwen could contribute a more informed look at the log.
If you want to try, drop the -n but add readonly
zpool import -fF -o readonly=on ISCSI
The errors seem to indicate that the root block pointer, rootbp, is damaged. Later it mentions labels are damaged. Neither is good because a RAID-Z2 should be able to survive 2 disks of failure, and even more redundant metadata failures without data loss.
Some of the ways for extreme corruption like this to occur:
In your case, it probably is the hardware RAID controller. But, you can check the firmware on the boot time messages. If it says something like MegaRAID and not IT, then it’s hardware RAID firmware.
Yes and no. If the problem is on the disks, regular ZFS scrubs can detect and fix it. But, if the corruption is happening during writes, like out of order writes, and then a power loss occurs before the final write is complete, well, bad things happen.
ZFS purposefully writes data and metadata in a specific order, using copy on write to free space. This means until the very last write, the changes are not active. However, an out of order write, like with hardware RAID card doing elevator writes, can activate garbage because the highest level block pointers, points to not yet written directory entries. And during a power loss, those lower level directory entries remain garbage.
Now a single power loss with out of order writes probably should not corrupt a pool. ZFS should be able to roll back transactions to something that is good. However, we always recommend rock stable hardware, (aka no hardware RAID controllers), just to avoid things like this.
You could try this below, and see what it has to say:
zpool import -FfXn ISCSI
Or you could look at all the disk labels using something like below. Then maybe temporarily remove a disk or 2 that appears radically different from the others. Mostly the first txg: number should be the same for all disks.
zdb -l /dev/sdX
To clarify for all, including future readers, ZFS should never corrupt a pool on power loss:
Hey thanks for the reply and the clarification.
i have had a look at the log and looks like it just keeps complaining that the the corrections/discards it needs to make are larger then possible. I’m not really an expert but i do like to understand what is happening. So correct me if I’m wrong 
I really like truenas for a lot of reasons but seems like there are a lot of gotchas when it comes to hardware. I went back to my original purchase and the raid card was sold to me as IT mode. I believe it was IT mode firmware but as many have suggested the raid card still looks like its doing something weird.
If i ran a -F import would I get anything back? or would it just fail at this point. If it’s not going to result in anything I’ll just pull and wipe my drives. Looking at moving to an off the shelf NAS unit. At least it will eliminate me as a factor with doing a weird hardware setup. Might mean I get a simpler overall setup.
log3.txt (136.8 KB)
Logs from zpool import -FfXn ISCSI
zdb -l /dev/sdX I get failed to upack label 0,1,2,3
was able to get storcli to work finally
storcli show all
CLI Version = 007.1207.0000.0000 Sep 25, 2019
Operating system = FreeBSD 13.1-RELEASE-p2
Status Code = 0
Status = Success
Description = None
Number of Controllers = 1
Host Name = truenas.home
Operating System = FreeBSD 13.1-RELEASE-p2
StoreLib IT Version = 07.1300.0200.0000
Ctl=Controller Index|DGs=Drive groups|VDs=Virtual drives|Fld=Failed
PDs=Physical drives|DNOpt=Array NotOptimal|VNOpt=LD NotOptimal|Opt=Optimal
Msng=Missing|Dgd=Degraded|NdAtn=Need Attention|Unkwn=Unknown
sPR=Scheduled Patrol Read|DS=DimmerSwitch|EHS=Emergency Spare Drive
Y=Yes|N=No|ASOs=Advanced Software Options|BBU=Battery backup unit
Hlth=Health|Safe=Safe-mode boot
There is a raid 1 on that controller but it is just for the boot drive from memory. nothing to do with ISCSI pool. I just wanted audible beeping from the controller if I have boot drive failure so I could change it out. Was the simplest way I could think of to do that. hence 11 drives 9 plus the two SSD’s in raid 1.
Might be part of the problem? but truenas is still booting fine.
IT mode does not allow any RAID, so the indication of one VD, Virtual Disk, (for your boot pool), is proof it is RAID. However, the BBU, Battery Backup Unit, is N/A so it implies MPT / IR style RAID firmware not the MegaRAID style. MPT / IR RAID does not support RAID-10, RAID-5/6/50/60 and should generally pass through unused disks fine. But, I don’t know much about the IBM version of the firmware.
The zdb -l /dev/sdX requires you to replace X with the drive letter, (and partition number if needed), of each of the ZFS pool disks. (But, not any import pool like the boot-pool…)
Last, the errors keep referring to a disk with a radically different TXG number. We need to see if that is 1 or 2 disks, (it appears just 1). If so, we can physically remove the disk and see if the improves the ability to import the pool.
Might be pulling a noob error here.
but by disks are da1 - da9
zdb -l /dev/sda1 give no file found
zdb -l /dev/da1 gives failed to unpack label
also i tired the with da2 and da9 same output.
You need to target the partition, not the drive.
Not this: /dev/da1
This: /dev/da1p2
If that is the correct paths to the ZFS disks and partitions for the affected pool, missing labels is REALLY BAD ™.
I don’t have the TrueNAS Core commands handy to determine which disks and partitions belong to which pools. That would help clarify your setup and check for use with ZDB.
Are you using encryption?
I dont believe so. That is not usually something i setup for home use.
It was orginally configured a long time ago. So i dont recall if i enabled encryption or not.
It is possible I dont have the correct patitions. If someone can help me with the commands to check the partitions that would be useful.
Thanks
I just tested this in a VM (TrueNAS-13.0-U6.7)
gpart show
dmesg might be of some use too
geli list should show encryption. In my case, it was just encrypting the swap.
I have pwenc_secret from when I created the truenas box backed up.
But it looks like its ony encrypting partition 1 the zfs partitions are all partition 2
Those are the encrypted swap partitions. You can tell by their 2GB size.
Is there anything else in the output? Your image is cropped.
