Pool degraded after update from 24.10.2.1 to 25.04.0

[winnielinnie]

@Captain_Morgan

I found one other instance in our bug tickets
Jira

That might have been a different issue or maybe similar but manifesting with a different and less severe outcome.

The real concern is not that the pools keep exporting, but that they cannot be imported again.

---

@Captain_Morgan

It looks like all the cases you mentioned were upgrading to 24.10. If we could confirm that it would be useful.

It gets more difficult to know the underlying issue, since one of the examples had skipped ahead and recovered their files using data recovery software. They have since formatted their disks with Windows, which makes it impossible to diagnose what happened.

Another was only able to get their pool imported by reverting to 23.10.x. I’m not sure how comfortable they feel trying again with an upgrade, which might result in the same or worse outcome.

The remaining two examples are inconclusive, and as of yet are still unable to import their pools.

It is indeed possible that the pattern is an upgrade to 24.10.x and not necessarily 25.04 (since the problem might have been triggered between the two upgrade steps). Maybe not even 24.10.x, but a specific minor version?

It’s also possible that it could equally be 24.10 or 25.04, and perhaps there’s a common trigger that only affects those two. (Maybe because of a change in partitioning? Something in OpenZFS?)

---

Does it warrant a new thread, so this one doesn’t get derailed?

I won’t be able to troubleshoot, and we’re still waiting on the results of the other two users.

It does seem to be a pattern that happened recently, and they all involved unimportable pools immediately after an upgrade.

It’s not certain yet, but it could also be triggered based on when and how the pool was created.

---

I could ask the other two users to direct message @HoneyBadger to send this:

@HoneyBadger

asking a couple folks to dd me the first 32MB of their drives/partitions, so I can carve 'em up with a hex editor and see if there’s a consistent pattern.

[Captain_Morgan]

@winnielinnie

It does seem to be a pattern that happened recently, and they all involved unimportable pools immediately after an upgrade.

It’s less likely to be “recent” than associated with a specific update (Could be x → ZFS 2.3)

In any case, it would be good to find a specific example that can be diagnosed. Preferably its a simple update and pool import issue… then make it a NAS-ticket we can trouble shoot.

My guess is there’s been about 100,000 updates to 24.10 and 25.04… so I don’t think its that common. A corner-case that we don’t understand?