S.O.S. Is my data gone? Pool drives appear as unassigned disks. Please Help!

It’s FIXED, everything is back to normal just as it were! :slight_smile: :tada: :tada:

HoneyBadger, I’d like to publicly thank you again for your prompt and expert help! This is easily the best support I ever experienced and it lights very good light on iXsystems, but also the wider TrueNAS community as well.

The solution was very short but specific to my system. HoneyBadger will maybe summarize it for future reference.

Now, for context and clarification, I’d like to react to this. You either didn’t read all that I wrote or misunderstood. HoneyBadger told me there were several unfortunate and unlikely factors that came into play, but there was no error on my part. I think that is enough in my defense.

In my first approach, I guessed, and in light of everything that went down since then, it seems to me that would have been the correct approach (although actually doing the pool expansion is still a task to be done). However, I didn’t go through with this approach, exactly because I wanted outside confirmation. This “guesswork” thus didn’t do any modification to my system. When I started to read manuals and the forums, that’s exactly when I found and tried the approach that ended in my pool falling apart. As far as I understand, it shouldn’t have happened btw, but that was the reason. Ironically, exactly RTFM was the culprit here, which is IMHO a golden example to what John has written.

Hurrah!!!

And as @Prusstalc says, a very well done to @HoneyBadger who has put in a lot of effort to recreate the issue and determine a solution!!!

As to the other, unrelated problems of my system…

The Purple was pulled from the mirror because I was about to destroy that pool just as I expand the other one. It was vastly quicker not to re-cable the server and work on it twice to install a new drive and remove another one. I had this pool as a temporary solution anyway because the 10TB one was filled. I had two (mismatched) 4T drives so I installed them as mirror because I wanted to try how to set up that.

The HE10 is alone in the pool because I didn’t have money for a buddy for it. As far as I understand, RAID is mainly for availability, not for security. I don’t really need availabilty, but I need security, that’s why I had backup of the important data. (That’s a different question that losing that 2TB that I originally deemed not so important, would have been a pain, too. And this made me realize I might want at least one backup of everything.)
I definitely planned to mirror the pool in a couple of months, though. It’s just that right now I needed more space, not redundancy. So I bought the Barracuda to have 18 TB. I planned to add an 18TB disk as mirror (or 10+8 again, if that’s not possible).

BTW, we are speaking about a couple months of “risky setup”, the system was installed about 2 months ago. I deemed this time span accaptable risk for a datacenter drive that isn’t even used that much.

About the SMR drives. I know (something) about the CMR-SMR difference, but I didn’t pay that much attention to it. This setup doesn’t do much, really. The data on it is important but mostly archive data that I write once and rarely read (if at all). There is some R/W daily but not much, 1-10GB. In time, I certainly will find other things for it to do, I have some ideas. For this limited use, the lower write performance of the SMR drive didn’t seem to be a big compromise, given that it’s vastly-vastly cheaper.

Is there any other restriction of an SMR drive that I didn’t take into consideration? I can be convinced. If it’s really unfit for TrueNAS then I’ll put it instead in the small Zyxel NAS that I use for backup only.

EDIT: Protopia wrote before that resilvering takes a lot of time on an SMR drive. Is it the case of mirror as well? I don’t ever plan to use parity RAID, definitely not in this setup.

Ah yes, an important lesson came to mind about the fix. For anyone in a similar situation: If you are doing any change to the partition table during a fix, always triple-check the drive path! My HE10 was sdb initially, but after a reboot it changed to sda. There wasn’t any recabling, HW change or SW change between the two states. It just changed somehow during a power-off/power-on cycle. I don’t understand why, maybe it has to do with 3 disks being unassigned at that time.

Just check the path!

When resilvering a disk it triggers the pathological worst case scenario for an SMR disk.

In the best case, this means that resilvering can take an order of magnitude longer, in the worst case, it can fail as the disk simply gives up and returns an error.

The rest of the time SMR disks are fine… it’s a bit insidious.

Here is my understanding of SMR drives…

SMR drives have VERY large blocksizes under the covers.

To write a 512 sector into a large block it needs to read the block, insert the new sector and write the block again. This makes native writes very slow.

To compensate for this, SMR drives have a CMR cache area where writes can be staged at normal speeds, and then the firmware transfers these sectors from the CMR cache into the SMR area in the background.

If you are writing sectors in bulk e.g. because you are doing a resilver, then the CMR cache gets full and then every additional write slows down to SMR speeds - so your resilvering takes forever and a day. This is what makes them unsuitable for any form of RAID / Mirroring especially ZFS.

But as you suggest, SMR disks are probably fine for non redundant usage.

Indeed they are probably fine for use in RAIDZ / mirrors which start empty just so long as you don’t ever need to resilver them. Of course, you do not want to get to the point that for some reason you need to resilver to it so probably best not to use SMR drives for redundant pools regardless.

…and of course you will need to resilver, if you keep your pool long enough.

Though, isn’t it only a problem when resilvering TO an SMR drive? So I should be fine as long as I take caution to buy a CMR drive as spare when a drive fails. Right? Then SMR drive(s) in the pool are only read during resilvering, only the new spare will be written to. If it’s CMR, everything is dandy.

As I see it, this is a solid and cheap approach. Build a pool with SMR drives, whenever one fails, put a CMR in place, and Bob’s your uncle.

Lmao

Except if you get a temporary error (e.g. because of a loose SATA cable) and the SMR drive drops out of the pool - then you have a working SMR drive but still have to replace it with a CMR one to get the pool working again.

For the sake of a pretty small amount of money, get a CMR drive from the start and save yourself some headaches later.

@HoneyBadger can you please share what caused the problem and how it got fixed? Thank you!

Basically what happened here was the partition table got an incorrect offset on @Prusstalc 's 10T disk, so ZFS didn’t know where to enter to pick up the labels. ZFS stores four copies of the label (two at the front, two at the end) but an incorrect offset basically results in an “off-by-one” style error so all four addresses missed the mark. I provided the correct partition table via PM to the OP who (after backing up the existing one) overwrote it, and was back in business. Because this information needs to be exact (including LBA count and partition GUIDs) I didn’t want someone to just grab a code snippet from here and blindly apply it to their disk(s) if they think they’re experiencing the same problem, because even if they are it won’t work for them.

On the always fun SMR subject, the challenge is that the devices effectively encounter a Read-Modify-Write cycle for anything smaller than 256 MiB. In ZFS terms, think about instead of ashift=12, they need to have ashift=28 - and then you bottleneck behind the CMR cache zone as outlined by @Protopia here

In a hypothetical future-state, Host Aware or Host Managed SMR drives might be viable through Zoned Block Commands (ZBC) - but it would require ZFS to maintain an indirection/LBA mapping table into the SMR zones, similar to how an SSD does LBA translation from logical LBA to a location on a physical NAND page, a “very non-trivial” amount of work would be involved with that.

For the time being, I treat SMR drives like a tape cassette with a SATA port welded onto them.

Are you referring to the “zfs_member” label that is per block device. (As in, it could be an entire drive, a partition, or even a file?)

If so, then what good is having four copies if ZFS is not smart enough to handle an “off-by-one” due to an incorrect offset?

Is it that ZFS dare not touch anything “partition table related” and thus, it won’t even try to detect a ZFS member device?

To use a different analogy, take testdisk for example. Without knowing anything about a drive, its partition table, its partitions, any filesystems, or what files might exist after a quick format, it can still recover a JPEG image simply by scanning for its header/signature.

You’d think in 2024, with ZFS’s sophistication, it could do the same thing for ZFS labels? Maybe something like: zpool find-label /dev/sda1


Anyways, since this issue has been resolved, can we have ourselves a good old-fashioned FreeBSD vs Linux debate? :pray:

A TrueNAS bug? Is it going to be fixed?

zhack label repair exists but as the command implies, zhack is kinda, well, hacky. When you have redundancy from other disks, it’s better to rebuild from that. It also takes a pretty darn long time for testdisk to scan disks. :wink: In this case, I recognized the symptoms here and went for the known fix.

We actually fixed something similar already in an earlier release, so we’ll close off the corner-case here and get it committed. :slight_smile:

This entire time I’ve had a tool called “zhack” at my disposal, made by the OpenZFS developers?

:astonished:

Here be dragons.