Unable to Import ZFS Pool Due to Missing Virtual Cache Devices

TrueNAS General
1

Hello,

My ZFS pool tank1 has become unavailable due to missing virtual cache devices. Below are the details of my current situation:
cannot import ‘tank1’: no such pool or dataset Destroy and re-create the pool from a backup source.

TrueNAS version: TrueNAS-13.0-U6.2

pool: tank1
  id: 2821495267620698100
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
    devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
 config:

        tank1                                           UNAVAIL  insufficient replicas
          raidz1-0                                      ONLINE
            gptid/1f36bbf8-ed6c-11ec-bcdf-000c293d6a54  ONLINE
            gptid/1f672568-ed6c-11ec-bcdf-000c293d6a54  ONLINE
            gptid/1f6a9da2-ed6c-11ec-bcdf-000c293d6a54  ONLINE
            gptid/1f68a515-ed6c-11ec-bcdf-000c293d6a54  ONLINE
            gptid/1f6a0d7d-ed6c-11ec-bcdf-000c293d6a54  ONLINE
          raidz1-1                                      UNAVAIL  insufficient replicas
            gptid/e355dde3-4f0a-11ee-899b-000c296b65a6  UNAVAIL  cannot open
            gptid/e359dadd-4f0a-11ee-899b-000c296b65a6  UNAVAIL  cannot open
            gptid/e357f651-4f0a-11ee-899b-000c296b65a6  UNAVAIL  cannot open

This problem arose after we exported the pool and try to import it again. Cache drives which are now missing from the system. These cache drives were part of the pool’s RAIDZ1 configuration. When we tried to import the pool back, the missing virtual cache drives caused the pool to be unavailable.
Steps Taken:

Attempted to import the pool using zpool import -f tank1 without success.
Tried to use partial import commands zpool import -F -X tank1 and zpool import -d /dev/gptid -o readonly=on -F -X tank1, but the pool remains unavailable.
Verified the physical connection of all drives.
Analyzed the pool configuration with zdb -e tank1, which indicated the missing devices but was unable to proceed further.

Debug Messages:

spa.c:6110:spa_import(): spa_import: importing tank1
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): LOADING
...
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): vdev tree has 1 missing top-level vdevs.
...
spa_misc.c:404:spa_load_failed(): spa_load(tank1, config trusted): FAILED: unable to open vdev tree [error=2]
...

here is zdb -e tank1

Configuration for import:
        vdev_children: 2
        version: 5000
        pool_guid: 2821495267620698100
        name: 'tank1'
        state: 1
        hostid: 1886089997
        hostname: 
        vdev_tree:
            type: 'root'
            id: 0
            guid: 2821495267620698100
            children[0]:
                type: 'raidz'
                id: 0
                guid: 16474386183726751154
                nparity: 1
                metaslab_array: 65
                metaslab_shift: 34
                ashift: 12
                asize: 39997054648320
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 8430242691483147646
                    DTL: 12595
                    create_txg: 4
                    path: '/dev/gptid/1f36bbf8-ed6c-11ec-bcdf-000c293d6a54'
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 8219385248927223757
                    DTL: 12594
                    create_txg: 4
                    path: '/dev/gptid/1f672568-ed6c-11ec-bcdf-000c293d6a54'
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 17813456083485363334
                    DTL: 12593
                    create_txg: 4
                    path: '/dev/gptid/1f6a9da2-ed6c-11ec-bcdf-000c293d6a54'
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 7474179783460172973
                    DTL: 12592
                    create_txg: 4
                    path: '/dev/gptid/1f68a515-ed6c-11ec-bcdf-000c293d6a54'
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 1345862483031176778
                    DTL: 12591
                    create_txg: 4
                    path: '/dev/gptid/1f6a0d7d-ed6c-11ec-bcdf-000c293d6a54'
            children[1]:
                type: 'missing'
                id: 1
                guid: 0
        load-policy:
            load-request-txg: 18446744073709551615
            load-rewind-policy: 2
zdb: can't open 'tank1': No such file or directory

ZFS_DBGMSG(zdb) START:
spa.c:6110:spa_import(): spa_import: importing tank1
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): LOADING
vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/1f36bbf8-ed6c-11ec-bcdf-000c293d6a54': best uberblock found for spa tank1. txg 12988230
spa_misc.c:419:spa_load_note(): spa_load(tank1, config untrusted): using uberblock with txg=12988230
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): vdev tree has 1 missing top-level vdevs.
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
spa_misc.c:404:spa_load_failed(): spa_load(tank1, config trusted): FAILED: unable to open vdev tree [error=2]
vdev.c:216:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 2821495267620698100, path: N/A, can't open
vdev.c:216:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 16474386183726751154, path: N/A, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 8430242691483147646, path: /dev/gptid/1f36bbf8-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 8219385248927223757, path: /dev/gptid/1f672568-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 17813456083485363334, path: /dev/gptid/1f6a9da2-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 7474179783460172973, path: /dev/gptid/1f68a515-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 1345862483031176778, path: /dev/gptid/1f6a0d7d-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():     vdev 1: raidz, guid: 6809027625265124393, path: N/A, can't open
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 16789085765231373816, path: /dev/gptid/e355dde3-4f0a-11ee-899b-000c296b65a6, can't open
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 3490384972256348933, path: /dev/gptid/e359dadd-4f0a-11ee-899b-000c296b65a6, can't open
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 11011393509591623693, path: /dev/gptid/e357f651-4f0a-11ee-899b-000c296b65a6, can't open
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): UNLOADING
ZFS_DBGMSG(zdb) END
root@storage[~]#
root@storage[~]# >....
                    id: 1
                    guid: 8219385248927223757
                    DTL: 12594
                    create_txg: 4
                    path: '/dev/gptid/1f672568-ed6c-11ec-bcdf-000c293d6a54'
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 17813456083485363334
                    DTL: 12593
                    create_txg: 4
                    path: '/dev/gptid/1f6a9da2-ed6c-11ec-bcdf-000c293d6a54'
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 7474179783460172973
                    DTL: 12592
                    create_txg: 4
                    path: '/dev/gptid/1f68a515-ed6c-11ec-bcdf-000c293d6a54'
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 1345862483031176778
                    DTL: 12591
                    create_txg: 4
                    path: '/dev/gptid/1f6a0d7d-ed6c-11ec-bcdf-000c293d6a54'
            children[1]:
                type: 'missing'
                id: 1
                guid: 0
        load-policy:
            load-request-txg: 18446744073709551615
            load-rewind-policy: 2
zdb: can't open 'tank1': No such file or directory

ZFS_DBGMSG(zdb) START:
spa.c:6110:spa_import(): spa_import: importing tank1
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): LOADING
vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/1f36bbf8-ed6c-11ec-bcdf-000c293d6a54': best uberblock found for spa tank1. txg 12988230
spa_misc.c:419:spa_load_note(): spa_load(tank1, config untrusted): using uberblock with txg=12988230
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): vdev tree has 1 missing top-level vdevs.
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
spa_misc.c:404:spa_load_failed(): spa_load(tank1, config trusted): FAILED: unable to open vdev tree [error=2]
vdev.c:216:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 2821495267620698100, path: N/A, can't open
vdev.c:216:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 16474386183726751154, path: N/A, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 8430242691483147646, path: /dev/gptid/1f36bbf8-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 8219385248927223757, path: /dev/gptid/1f672568-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 17813456083485363334, path: /dev/gptid/1f6a9da2-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 7474179783460172973, path: /dev/gptid/1f68a515-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 1345862483031176778, path: /dev/gptid/1f6a0d7d-ed6c-11ec-bcdf-000c293d6a54, healthy
vdev.c:216:vdev_dbgmsg_print_tree():     vdev 1: raidz, guid: 6809027625265124393, path: N/A, can't open
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 16789085765231373816, path: /dev/gptid/e355dde3-4f0a-11ee-899b-000c296b65a6, can't open
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 3490384972256348933, path: /dev/gptid/e359dadd-4f0a-11ee-899b-000c296b65a6, can't open
vdev.c:216:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 11011393509591623693, path: /dev/gptid/e357f651-4f0a-11ee-899b-000c296b65a6, can't open
spa_misc.c:419:spa_load_note(): spa_load(tank1, config trusted): UNLOADING
ZFS_DBGMSG(zdb) END

Thank you for your assistance.

Juha

2

No sure what you mean by this. By the looks of your pool your have one vdev that’s a 5 disk Z1 and another that’s a 3 disk Z1 which is rather unusual. More to the point the 3 disk vdev appears to be missing all 3 drives.

Perhaps a zpool status (that isn’t chopped as much) from the command line may help a bit more to clarify.

1 Like
3

Hello,

I apologize for my poor explanation earlier. Here is the situation: I was trying to remove the RAIDZ1-1 section, which was serving as a cache in the tank1 pool. The SSD in that RAIDZ1-1 section broke. I read instructions that suggested exporting the pool, removing the damaged SSD, and then importing the pool again to remove the RAIDZ1-1 section, which clearly did not work. :frowning:

However, now I cannot import the pool because the drive is no longer available. All data should be on the RAIDZ1-0 drives. Is there any way to remove the RAIDZ1-1 section and then import tank1 again?

I hope this clarifies the issue. :slight_smile:

Thank you for your help.

 pool: tank1
 id: 2821495267620698100

state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: Message ID: ZFS-8000-3C — OpenZFS documentation
config:

    tank1                                           UNAVAIL  insufficient replicas
      raidz1-0                                      ONLINE
        gptid/1f36bbf8-ed6c-11ec-bcdf-000c293d6a54  ONLINE
        gptid/1f672568-ed6c-11ec-bcdf-000c293d6a54  ONLINE
        gptid/1f6a9da2-ed6c-11ec-bcdf-000c293d6a54  ONLINE
        gptid/1f68a515-ed6c-11ec-bcdf-000c293d6a54  ONLINE
        gptid/1f6a0d7d-ed6c-11ec-bcdf-000c293d6a54  ONLINE
      raidz1-1                                      UNAVAIL  insufficient replicas
        gptid/e355dde3-4f0a-11ee-899b-000c296b65a6  UNAVAIL  cannot open
        gptid/e359dadd-4f0a-11ee-899b-000c296b65a6  UNAVAIL  cannot open
        gptid/e357f651-4f0a-11ee-899b-000c296b65a6  UNAVAIL  cannot open
4

So firstly to me that missing vdev doesn’t look like a cache (ie L2ARC or even SLOG) it looks like a data vdev.

Secondary where have all those three drives gone?

5

Thank you for your response.

The RAIDZ1-1 section was intended to be used as a cache and was set up as L2ARC. These drives have been functioning for a year without issues until they recently failed due to hardware issue.

6

Please post your entire hardware, software and pools setup. The specifics make a difference.
You can expand my listing in post for an example of what we are looking for.

7

Cache vdevs appear like this:

Screenshot 2024-07-14 at 18.29.38

Yours was a data vdev. Why three of the drives have vanished is the real question here. As suggested please provide all system specs.

2 Likes
8

That might have been your intent, but you system presents it differently.
It really does look like the missing vdev was a basic data vdev, not a cache vdev.

If that is the case, and those drives are no longer available, you’ll looking at a total pool loss. Do you have a backup should that turn out to be the case?

Also, to clarify, you’ve used the word “virtual” in your post, is your TrueNAS install running as a VM? If so, what host OS and please elaborate on how you linked the drives to the VM; virtual disks tied to the VM, pass-through of physical disks or something else, like pass-through of an HBA or similar?

1 Like
9

Cannot do that. Raidz# vdevs cannot be removed.

Does not work either. ZFS doesn’t support tiered storage.

No and no. If the SSD have not totally failed, put them back. These drives hold data, and without them the pool is lost.

1 Like
10

Need to restore at least 2 of these drives.

If you can not, the pool is lost.

11

To clarify what others have said, L2ARC, aka “Cache”, vDev(s) can not be RAID-Zx, nor Mirrored. They are either single disk or multiple disks in a stripe. Cache / L2ARC vDevs are non-critical, and loss of one or more is potentially a performance issue, not a data loss issue.

You’ve implied that the “raidz1-1” is a “Cache” vDev type. As others have pointed out, this is impossible. It is beyond any doubt a data vDev, which is critical to the pool.

12

Maybe they thought that adding SSDs to a HD pool would be “cash” :scream: