I had a pool with a single 10TB disk as data vdev and a 256GB SSD as L2ARC. I was running very low on space, only had 78GB free space left, so I bought a 8TB disk on sale and plugged it in. I wanted to grow the pool to 18TB but I found conflicting instructions on how to do that.
Someone suggested clicking on “Expand” on the Storage page so I did that. The operation was not successful, I got an error with the a suggestion to restart the system immediately. It didn’t seem very serious to me, I didn’t expect a destructive operation and it didn’t go through and I expected such operation to be transactional (success or no change). I didn’t read the error, I only saw that it was something about the partition table not being rewritten and falling back to original (at least that’s how I understood it in the 2 seconds looking at it). I followed the recommendation of rebooting. And now the pool is ruined. It shows “Offline VDEVs” and my data disk and the L2ARC SSD are both listed among the unassigned disks along with the new 8TB disk.
The pool was encrypted, I have the pool decryption key saved though.
How screwed am I? Is the data completeley lost?
I have backup of the most important things but not “mildly important” things. I know a striped pool isn’t safe to start with but I didn’t have money for mirror yet. I planned to add a mirror drive later, now I needed extra storage. Not that this has anything to do with the issue, just for context.
EDIT: I tried to find the error log but couldn’t find it. I didn’t find any logs on the UI and the system/kernel logs I was able to view in terminal didn’t contain anything related to the issue. At least I couldn’t find it. Is there a place to find such error logs?
I have (had) two pools. First pool was, as described:
1 HGST HE10 10TB drive as data
1 250GB Samsung SSD as L2ARC
Second pool (mirrored) is:
1 4TB WD Purple as data (removed)
1 4TB WD Blue as data
And the new 8TB Seagate Barracuda, which is plugged in but not used so far.
The server is on a 1 Gbit LAN, only locally used so far, not allowed connection from the internet.
(Today, when I installed the new drive, I removed the WD Purple because I was planning to move the little data that pool contained to the first pool, remove the second pool and reuse the drives elsewhere. The second pool is shown as degraded, but functional and I don’t think this is relevant to the main issue.)
Did you try zpool online poolname ?
I didn’t. Should I try it?
What is the output of zpool status poolname?
It says “no such pool”. But I still see it in the UI. I don’t get it.
I suspect your pool has gone. But I am not an expert.
At this point you MUST NOT DO ANYTHING unless you are sure that it is the right command. If your data is still recoverable, then doing the wrong thing will likely make it unrecoverable.
As far as I can tell ZFS is a VERY safe file system if it is administered properly, but because pools can be VERY large and because a pool is a single unit of storage it is essential that you know what you are doing when administering it because you can lose EVERYTHING if you get things wrong.
So if anyone gives you advice on how to recover it (myself included - or indeed especially if it is me ) get a 2nd opinion and do your own research to check whether it sounds like the right thing to do before you do anything.
Also you need to check whether your WD Blue 4TB is SMR or CMR technology - please provide the exact model number - because SMR technology is unsuitable for redundant drives because its write performance during resilvering will be VERY poor.
Thanks, yes, I got that info from the emulator. It’s just that I don’t have an insert key. I generally don’t recognize it as being useful and I remapped it to be my compose key which has more utility for me. This is the first instance in my life when I actually needed the insert key.
You may want that functionality for these steps, we’ll be doing a bit of text here.
Can you show me the same output for your WD Blue in the mirrored pool, as well as the Swap Size value under System Settings → Advanced and scroll down to the Storage pane
Can you do the sfdisk output against the new 8T disk? I’m curious as to if it was successfully added to the pool or if you just attempted an Expand operation against the single-disk 10T
) get a 2nd opinion and do your own research to check whether it sounds like the right thing to do before you do anything.
I generally don’t recognize it as being useful and I remapped it to be my compose key which has more utility for me. This is the first instance in my life when I actually needed the insert key.