ZFS Pool disappeared after IO error, no longer listed

TrueNAS General
,
1

Hi all, I was performing some disk benchmarks this afternoon using dd and fio when my entire zfs pool went offline with the following error:

  • Pool nuhtshack state is SUSPENDED: One or more devices are faulted in response to IO failures.
    The following devices are not healthy:
    • Disk WDC_WUH721816ALE6L4 2CHHXGXJ is UNAVAIL
    • Disk WDC_WUH721816ALE6L4 3WHHAE5K is UNAVAIL
    • Disk WUH721816ALE6L4 3HHBW3SN is UNAVAIL

When trying to import, the pool seems to be nowhere to be found. zpool import shows ‘no pools available to import’, it’s not listed in zpool status. Nothing I saw indicated drive failure, SMART data looked healthy, the drives only have about 900 hours on them. The drives themselves are still detected. Is there any way to recreate the pool without wiping the data on them?

2

Are you running TrueNAS in a VM?

3

Uh… seriously? You’re running dd and fio benchmarks on a live pool!??!?
Do you not know what these things do? They are supposed to be destructive you know (especially dd)… Frankly, I would be more surprised if your pool DIDN’T go missing cause that means those tools didn’t really do their jobs.

You sound eerily like this guy… Same culprits, dd and fio, and even started the article with similar sentence.

So, i was benchmarking the high end hardware for cloud computing system that im working on.

4

nope, I guess I didn’t fully understand it was destructive. Well now I know for the future, thanks

5

Well, you know how the saying goes.

Never let a good disaster go to waste.

You should take the lesson now that you should never just blindly copy/paste commands you see over the internet without fully understanding what it is actually doing or without asking/confirming with others who do know.

1 Like
6

It may still be worth it to try to see if you can salvage the pool and maybe one of us here can help. Listing your HW configuration and how the disks are connected would be a start.

Also, post the outputs of lsblk and fdisk -l for starting point.

7

Oh it’s alright, I accepted defeat about an hour ago and am starting fresh. I mainly just wanted to know why it happened so I can avoid doing the same thing in the future. Appreciate it though

1 Like