ZFS ZPool Import Fails

TrueNAS General
, ,
1

I recently issued a poorly executed sgdisk -Z command to my primary data pool and which caused it to offline after the next reboot. In the Storage tab, there are no importable pools, however when in the page to create a new pool it warns me there are some disks with an existing pool that will be overwritten.

Here is the output of sudo zpool import:

admin@truenas[~]$ sudo zpool import
pool: Home Server
id: 6846509301544695995
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: Message ID: ZFS-8000-EY — OpenZFS documentation
config:

    Home Server                               UNAVAIL  insufficient replicas
      mirror-0                                ONLINE
        24ba140c-07aa-45b9-956c-7904cba579ac  ONLINE
        2e93bc7a-3d01-4913-9e74-0d2b183093ba  ONLINE
      mirror-2                                UNAVAIL  insufficient replicas
        7ee887ee-449b-457e-8ab0-ad4fced01937  UNAVAIL
        c3b8b4e2-4a9e-45eb-90a6-3fdf2eebb844  UNAVAIL
    logs
      747a027f-94bc-46e7-b952-c2d8662727b8    ONLINE

Mirror-2 would have limited data stored there, and I’m hopeful I can import mirror-0 to recover my data, given mirror-0 and the log disk are apparently online. Before I screw this up further, I was curious if anyone has any input.

2

If you created a checkpoint immediately prior to adding mirror-2, you could theoretically just use the --rewind-to-checkpoint option when importing the pool. (You will lose everything created / modified after the time of the checkpoint’s creation.)

Otherwise, it’s looking like your options are:

  1. A “recovery” import, with TXG magic, if it’s even possible
  2. Use a tool for ZFS data recovery, such as Klennet
  3. Restore from your most recent backup
3

ZFS does not work like you think. If you loose a data vDev, (aka Mirror-2), you loose the ENTIRE pool. Even if all the data was originally written to another working vDev, That said, recovery of some data may be possible using the previously mentioned Klennet tool.

So, you want to get back at least 1 of the disks in Mirror-2.

Now if you truly only zapped the MBR & GPT partition tables, it may be possible to re-create them. Tricky. This is what I would do;

  • Make a RAW backup of one of Mirror-2 disks
  • Guess what would be correct for the partition table, perhaps using Mirror-0 as an example
  • Use that information to re-create the partition table on the backed up Mirror-2 disk
  • Use sudo zpool import again and see if it worked
  • Repeat guesses for partition tables until you either succeed, or just want to give up
  • If you succeed, you then make a backup of the other Mirror-2 disk, and implement the same partition table.
  • Try sudo zpool import again, and if it appears good, do a real import
  • Run a scrub and see if all is good.
1 Like
4

Thank you both, I really appreciate it. I did not manually make a checkpoint so unless that’s an automatic process I don’t think it’s likely that I have one.

Regarding recreating partition tables, I’ve been looking into this for a few days and across the board that seems like the best possible solution. I’ll look into Klennet as well, I haven’t heard of that. Do you have any recommendations to start rebuilding the MBR and GPT?

Finally, for anyone else who finds this thread: I issued the sgdisk -Z command because I added the mirror-2 drives with the intention of creating a mirrored vdev, but truenas was refusing to let me add them to the pool. I don’t remember why, but I think it was spitting out errors about the disks already being used. However, I know for a fact that I issued the command to the new drives using the sda or sdb device identifiers provided by the web GUI. I would not recommend doing this, because despite the errors TrueNAS gave me, it seems the disks were actually being added to the pool and I caused this issue myself by being impatient.

5

DO NOT MAKE ANY FURTHER ATTEMPTS TO BRING THIS ONLINE OR RECOVER IT until a definitive understanding is reached about what caused it to be broken, how it is broken, whether all or even any data is recoverable and if so exactly how to achieve it. Doing something else can potentially make it worse and destroy any remaining chances of recovering data.

My reading of the output from the zpool import command is as follows:

  1. The Home Server pool consists of 4 drives, consisting of 2 vDevs each of which is a mirror. One of the mirror vDevs is fine and mirrored, but the other vDev is toast i.e. both drives are unavailable i.e. the ZFS partitions on those disks are unavailable. For ANY of the data in this pool to be recoverable, both vDevs needs to be available.

  2. I am unclear what the logs line is for - is this an SLOG for the Home Server pool?

  3. I am unclear which drive(s) you issued the sgdisk -Z against, but my understanding is that this command wipes the GPT Partition table for those disks and leaves them appearing empty. You have not stated why you issued an sgdisk -Z nor against which device, nor why you think it was poorly executed, nor is it clear why there are 2 disks where partitions are unavailable. We need to understand why both disks in the 2nd vDev mirror are shown unavailable.

  4. To bring this pool back online, it seems to me that you need to bring the 2nd vDev back online, so that is what we need to focus attention onto.

The immediate thought that comes to mind is to use sgdisk or gdisk to:

  1. Diagnose the state of partition tables on the unavailable disks;
  2. Attempt to recover any wiped partition table(s) using gdisk
  3. Then see what the zpool import now says.

However I am not a ZFS expert (though I have some recent experience with gdisk / sgdisk diagnosing and recovering partition tables), so hopefully some other community members will contribute their advice and let’s try to reach consensus on what you should do.

In the mean time please do the following:

  1. Give details of your disk hardware

  2. Run the following commands and paste the results here:

    • lsblk -o PATH,PARTUUID
    • sudo sgdisk -p /dev/sda etc. for each base device listed in lsblk
    • sudo zpool status -v so we can see what pools are online
2 Likes
6

@arwen 's suggestions may be valid, but at this stage I think that more diagnostic information would help us avoid doing anything which might make it worse or lose diagnostic information, and help us give precise instructions which might have a real chance of recovery.

7
  1. Home Server was originally two mirrored drives, probably mirror-0. I added two new mirrored drives to the pool, likely mirror-2, then issued the sgdisk command.
  2. logs is a SLOG drive for Home Server. It was striped but was alone; no other drives in SLOG.
  3. I issued sgdisk -Z against the two newest drives, again, I’m presuming that would be mirror-2. I issued sgdisk -Z because I found a forum post that mentioned it’s usefulness.
  4. I agree, I think given the statuses in the output of sudo zpool import, bringing mirror-2 back online seems like the logical solution.

admin@truenas[~]$ lsblk -o PATH,PARTUUID
PATH PARTUUID
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdc1 50476f61-c094-44a2-b45a-e2003a311ffa
/dev/sdc2 b76f53e8-6c62-424f-aed1-13d31f48ada7
/dev/sdd
/dev/sdd1 2c2940bd-0846-4fba-8349-73f4925fd029
/dev/sdd2 24ba140c-07aa-45b9-956c-7904cba579ac
/dev/sde
/dev/sdf
/dev/sdf1 ae3ba724-f5a3-4ffe-8ec8-8e0f50841b85
/dev/sdf2 2e93bc7a-3d01-4913-9e74-0d2b183093ba
/dev/sr0
/dev/mapper/sdc1
/dev/nvme1n1
/dev/nvme1n1p1 67506cb4-20e6-4e94-91c6-6d51b26a3937
/dev/nvme1n1p2 e4de308a-cd89-432c-b0cc-2abd1d3da78a
/dev/nvme1n1p3 3587948d-749d-4546-b5a0-976fee7ed34d
/dev/nvme0n1
/dev/nvme0n1p1 747a027f-94bc-46e7-b952-c2d8662727b8

admin@truenas[~]$ sudo sgdisk -p /dev/sda
[sudo] password for admin:
Creating new GPT entries in memory.
Disk /dev/sda: 11721045168 sectors, 5.5 TiB
Model: ST6000VN001-2BB1
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): FA7377C5-A6D2-4E73-9BF5-549F16E62CB6
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721045134
Partitions will be aligned on 2048-sector boundaries
Total free space is 11721045101 sectors (5.5 TiB)

Number Start (sector) End (sector) Size Code Name

admin@truenas[~]$ sudo sgdisk -p /dev/sdb
Creating new GPT entries in memory.
Disk /dev/sdb: 11721045168 sectors, 5.5 TiB
Model: ST6000VN001-2BB1
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 4B1652AE-F05E-4FD6-818D-3558EFECBAB7
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721045134
Partitions will be aligned on 2048-sector boundaries
Total free space is 11721045101 sectors (5.5 TiB)

Number Start (sector) End (sector) Size Code Name

admin@truenas[~]$ sudo sgdisk -p /dev/sdc
Disk /dev/sdc: 976773168 sectors, 465.8 GiB
Model: Samsung SSD 860
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): ED9C155F-295A-4E78-8C30-7148F5F2DA27
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 976773134
Partitions will be aligned on 2048-sector boundaries
Total free space is 4075 sectors (2.0 MiB)

Number Start (sector) End (sector) Size Code Name
1 2048 4196352 2.0 GiB 8200
2 4198400 976773120 463.8 GiB BF01

admin@truenas[~]$ sudo sgdisk -p /dev/sdd
Disk /dev/sdd: 15628053168 sectors, 7.3 TiB
Model: ST8000NT001-3LZ1
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 404D7508-89F2-4FFA-9684-FC083CB300F3
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 15628053134
Partitions will be aligned on 2048-sector boundaries
Total free space is 4715 sectors (2.3 MiB)

Number Start (sector) End (sector) Size Code Name
1 2048 4196352 2.0 GiB 8200
2 4198400 15628052480 7.3 TiB BF01

admin@truenas[~]$ sudo sgdisk -p /dev/sde
Creating new GPT entries in memory.
Disk /dev/sde: 11721045168 sectors, 5.5 TiB
Model: ST6000VN001-2BB1
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 553CC63C-73CC-4D8A-A2E3-AE79E50DB56E
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 11721045134
Partitions will be aligned on 2048-sector boundaries
Total free space is 11721045101 sectors (5.5 TiB)

Number Start (sector) End (sector) Size Code Name

admin@truenas[~]$ sudo sgdisk -p /dev/sdf
Disk /dev/sdf: 15628053168 sectors, 7.3 TiB
Model: ST8000NT001-3LZ1
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 25DBF31F-9507-48A5-8D24-FFD08A2E8BDB
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 15628053134
Partitions will be aligned on 2048-sector boundaries
Total free space is 4715 sectors (2.3 MiB)

Number Start (sector) End (sector) Size Code Name
1 2048 4196352 2.0 GiB 8200
2 4198400 15628052480 7.3 TiB BF01

admin@truenas[~]$ sudo sgdisk -p /dev/nvme0n1
Disk /dev/nvme0n1: 231270400 sectors, 110.3 GiB
Model: INTEL SSDPEK1A118GA
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 4C96FDF1-0442-4B02-8A1E-68A3ABF3BD67
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 231270366
Partitions will be aligned on 2048-sector boundaries
Total free space is 6076 sectors (3.0 MiB)

Number Start (sector) End (sector) Size Code Name
1 4096 231268352 110.3 GiB BF01

admin@truenas[~]$ sudo sgdisk -p /dev/nvme1n1
Disk /dev/nvme1n1: 28131328 sectors, 13.4 GiB
Model: INTEL MEMPEK1J016GAH
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): E61723A3-D313-4FC0-A621-045C4D62B1E3
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 28131294
Partitions will be aligned on 2048-sector boundaries
Total free space is 4062 sectors (2.0 MiB)

Number Start (sector) End (sector) Size Code Name
1 4096 6143 1024.0 KiB EF02
2 6144 1054719 512.0 MiB EF00
3 1054720 28131294 12.9 GiB BF01

admin@truenas[~]$ sudo zpool status -v
pool: Apps
state: ONLINE
config:

    NAME                                    STATE     READ WRITE CKSUM
    Apps                                    ONLINE       0     0     0
      b76f53e8-6c62-424f-aed1-13d31f48ada7  ONLINE       0     0     0

errors: No known data errors

pool: boot-pool
state: ONLINE
config:

    NAME         STATE     READ WRITE CKSUM
    boot-pool    ONLINE       0     0     0
      nvme1n1p3  ONLINE       0     0     0

errors: No known data errors

8

Guess you just need to set the gpt table to match what it would’ve been, and then set the partid of the zfs partition to be what it should be

Anyone know how to force a partid?

9

1./3. I guess the lesson to be learned here is not to blindly follow someone else’s suggestions without knowing that a) it is exactly the same situation as yours and b) you know what the command is going to do.

2.-An SLOG may or may not be useful depending on what you are doing with the pool. For example, if you are accessing it only over SMB from Windows it will likely do literally nothing.

Can you please tell us which devices were the ones you issued the sgdisk -Z against?

I will take a look at the detail of your posts in a couple of hours once I have time and see if I can work out what sgdisk or gdisk commands you need to issue to to recreate the partition table manually. Please bear in mind what I just said - don’t blindly follow my instructions until you understand what they are going to do and someone else has validated them.

I think it would be best to only bring one of the disks in the broken mirror vDev back - because if we get it wrong we then have the other disk to try. Once you have a degraded mirror back online, you can then resilver the other disk (which should be fast because there should be very little data on the new mirror pair).

10

Analysing the disk layout now. It looks like SDA/SDB/SDE are new 6TB Ironwolf disks whilst SDD/SDF are the existing 8TB Ironwolf mirror-0 disks.

We need to know which two of SDA/SDB/SDE were used for mirror-2.

Also in order to recreate the partition table for mirror-2, we need to know whether there was a 2GB swap file on the new disks. What version of TN was the original pool created under and what version are you using now (i.e. when you created the mirror-2 vDev)? Also we need to know what your default swap size setting is now (i.e. when you created the mirror-2 vDev) under System Settings / Advanced / Storage?

11

ZFS partitions do not use the whole drive, there is a gap between the last ZFS sector and the end of the disk - we need to work out what that is.

Since you have 3x 6TB Ironwolfs of which only two were used for mirror-2, providing we can identify which drive was not used then perhaps the easiest way to create the correct partition table would be to create a new single drive pool on this completely unused disk, and use the partition layout for that as the basis for recreating the partition table(s) on the mirror-2 disks.

We could even back up the temp partition table from this disk, copy it to one of the mirror-disks and tweak the UUIDs.

2 Likes
12

Is this screenshot data enough for us to know for certain which disks need to be adjusted?

Screenshot from 2024-06-07 18-11-13
Screenshot from 2024-06-07 18-11-131027Ă—448 18.4 KB

13

No - it tells me what disks are currently in the non-importable Home Server pool - it doesn’t tell us which disks you actually used for mirror-2 or which disks you zapped with sgdisk -Z.

You added the 2nd vDev, and you zapped the partition tables - what devices did you do this to?

14

Unfortunately I do not think we’ll be able to go off my memory. When I added the three new disks, I remember thinking it was odd that they were not added consecutively to the end of the list of devices (as in, I thought it was odd they were not added as devices SDE, SDF, and SDG, for example). Meaning the fact that we now have devices SDA, SDB, and SDE, contradicts what I recall. When I added them, I don’t think they used the same device identifiers as what is now being shown. Could we use sudo sgdisk -i or sudo sgdisk -L to output disk info that would show zapped partition tables?

15

No - sgdisk -L or -i only give more information about what is actually in the existing partition table, not information about partitions no longer listed.

I will do a bit more research to see whether there is another means of recovering partition table data.

16

Thank you @Protopia. I’d love to donate some cash if you’re successful in helping me avoid days of data regeneration. You’ve been very courteous with me.

17

Mind that drive letters may change at each reboot, as the kernel enumerates devices anew. Gptid and serial numbers are persistent.

18

Good to know, thank you. I’d still need a way of identifying which disk was affected by sgdisk -Z, but identifying the disk by serial number would probably make this much simpler.

19

I will do my best to help with whatever time I can give - and I would like you to donate whatever cash you think this is worth to a charity (ideally one I will nominate).

I will get back to researching this a little later today - my wife is recovering in hospital from a planned operation, so I am not able to make this research my highest priority.

1 Like
20

Ok - we have no idea which drive was never used, so we should try a tool to find partitions first in the hope that this will avoid us needing to overwrite a drive.

Since the most recommended tool “testdisk” is not part of TrueNAS, we should probably try parted which is installed as standard.

NOTE: The following instructions are based only on research and a guess - I have NOT tried this on my system since I don’t have an empty drive to try it on.

  1. Because we are going to do some very long running commands, either use the GUI shell and tmux or open a SSH session.

  2. Assuming that the device names haven’t changed, type parted /dev/sda and then p to list partitions and check that there aren’t any.

  3. Then try rescue 2K 6T and then wait a log time whilst it reads the entire drive to see if it finds any partitions. If it does, allow it to recover the partition(s) - the ZFS partition and possibly a swap file. If it doesn’t try /dev/sdb or /dev/sde.

If you find a ZFS partition, then we need more diagnostics to see what the UUID is and reset it if needed before we then try an import.