S.O.S. Is my data gone? Pool drives appear as unassigned disks. Please Help!

I’m in big trouble.

I had a pool with a single 10TB disk as data vdev and a 256GB SSD as L2ARC. I was running very low on space, only had 78GB free space left, so I bought a 8TB disk on sale and plugged it in. I wanted to grow the pool to 18TB but I found conflicting instructions on how to do that.

Someone suggested clicking on “Expand” on the Storage page so I did that. The operation was not successful, I got an error with the a suggestion to restart the system immediately. It didn’t seem very serious to me, I didn’t expect a destructive operation and it didn’t go through and I expected such operation to be transactional (success or no change). I didn’t read the error, I only saw that it was something about the partition table not being rewritten and falling back to original (at least that’s how I understood it in the 2 seconds looking at it). I followed the recommendation of rebooting. And now the pool is ruined. It shows “Offline VDEVs” and my data disk and the L2ARC SSD are both listed among the unassigned disks along with the new 8TB disk.

The pool was encrypted, I have the pool decryption key saved though.

How screwed am I? Is the data completeley lost? :frowning:

I have backup of the most important things but not “mildly important” things. I know a striped pool isn’t safe to start with but I didn’t have money for mirror yet. I planned to add a mirror drive later, now I needed extra storage. Not that this has anything to do with the issue, just for context.

EDIT: I tried to find the error log but couldn’t find it. I didn’t find any logs on the UI and the system/kernel logs I was able to view in terminal didn’t contain anything related to the issue. At least I couldn’t find it. Is there a place to find such error logs?

Did you try zpool online poolname? What is the output of zpool status poolname?

Please list your full hardware specs.

A few points:

  1. Expand would normally add a mirror. You cannot add an 8TB mirror to a 10TB disk - so that wouldn’t work.
  2. Your L2ARC is almost certainly wasted / not of any use to you
  3. What happens if you type zpool import at the command line? [or possibly su zpool import]

List your hardware

Thanks for helping!

My HW spec:
CPU: AMD Ryzen 3 2200G
Motherboard: Gigabyte B450 AORUS M AM4
RAM: 2x8GB G.Skill Aegis 3000
Boot drive: 500GB Samsung SSD

I have (had) two pools. First pool was, as described:
1 HGST HE10 10TB drive as data
1 250GB Samsung SSD as L2ARC

Second pool (mirrored) is:
1 4TB WD Purple as data (removed)
1 4TB WD Blue as data

And the new 8TB Seagate Barracuda, which is plugged in but not used so far.

The server is on a 1 Gbit LAN, only locally used so far, not allowed connection from the internet.

(Today, when I installed the new drive, I removed the WD Purple because I was planning to move the little data that pool contained to the first pool, remove the second pool and reuse the drives elsewhere. The second pool is shown as degraded, but functional and I don’t think this is relevant to the main issue.)

Did you try zpool online poolname ?

I didn’t. Should I try it?

What is the output of zpool status poolname?

It says “no such pool”. :frowning: But I still see it in the UI. I don’t get it.

What happens if you type zpool import at the command line?

“no pools available to import”. :frowning:

I suspect your pool has gone. But I am not an expert.

At this point you MUST NOT DO ANYTHING unless you are sure that it is the right command. If your data is still recoverable, then doing the wrong thing will likely make it unrecoverable.

As far as I can tell ZFS is a VERY safe file system if it is administered properly, but because pools can be VERY large and because a pool is a single unit of storage it is essential that you know what you are doing when administering it because you can lose EVERYTHING if you get things wrong.

So if anyone gives you advice on how to recover it (myself included - or indeed especially if it is me :grinning:) get a 2nd opinion and do your own research to check whether it sounds like the right thing to do before you do anything.

(See the bubble next to my name on these posts!!)

P.S. With only 16GB of RAM, a L2ARC will likely reduce performance rather than speeding it up.

If it turns out that your data is unrecoverable, I would suggest that you get some advice on how to configure your disks properly before rebuilding.

Also you need to check whether your WD Blue 4TB is SMR or CMR technology - please provide the exact model number - because SMR technology is unsuitable for redundant drives because its write performance during resilvering will be VERY poor.

That Seagate baracuda drive is to 99% also an smr drive

1 Like

@Prusstalc what version of TrueNAS are you currently running?

It’s Dragonfish-24.04.0.

Please identify your 10T He10 drive (Storage → Disks) and then issue the command

sudo sfdisk -d /dev/sdX

where sdX is the device that is your He10

Post the output in codeblocks (three backticks) here if possible.

1 Like

Sorry, copying doesn’t work from the Truenas shell, so I attach a screenshot:

To copy from the Truenas shell, highlight the info to copy then use ctrl-insert to copy. Use shift-insert to paste into the shell.

Thanks, yes, I got that info from the emulator. It’s just that I don’t have an insert key. :slight_smile: I generally don’t recognize it as being useful and I remapped it to be my compose key which has more utility for me. This is the first instance in my life when I actually needed the insert key.

You may want that functionality for these steps, we’ll be doing a bit of text here. :stuck_out_tongue:

Can you show me the same output for your WD Blue in the mirrored pool, as well as the Swap Size value under System Settings → Advanced and scroll down to the Storage pane

Alright, Chief, I disabled the compose key. :slight_smile:

WD Blue:

label: gpt
label-id: 66E39DF6-0DD5-42BF-882E-8C5600874EAB
device: /dev/sdd
unit: sectors
first-lba: 34
last-lba: 7814037134
sector-size: 512

/dev/sdd1 : start=        4096, size=  7814031361, type=6A898CC3-1DD2-11B2-99A6-080020736631, uuid=3FBFDA50-62AF-4D20-9BC4-AF00BFA797EA

System Dataset Pool:
Swap Size:
2 GiB

EDIT: Adding the He10 output again in text:


label: gpt
label-id: 367854F3-EBFE-4253-8EC8-1CDD92A85D9B
device: /dev/sdb
unit: sectors
first-lba: 6
last-lba: 2441609210
sector-size: 4096

/dev/sdb1 : start=        4096, size=  2441605115, type=6A898CC3-1DD2-11B2-99A6-080020736631, uuid=35A2110D-856F-4A19-904F-26D411D4A1B4

Thanks, trying to repro it now, but got interrupted by a power outage.

1 Like

Can you do the sfdisk output against the new 8T disk? I’m curious as to if it was successfully added to the pool or if you just attempted an Expand operation against the single-disk 10T

You mean “sfdisk”, right?
It seems to me, it is still in original, raw state:

admin@truenas[~]$ sudo sfdisk -d /dev/sda
sfdisk: /dev/sda: does not contain a recognized partition table

It’s sda because it went in the physical place of the Purple.