TrueNAS Core: Pool Offline and Disks automatically removed from the Pool

[maltafk]

I gave it a few hours, but I believe nothing is happening. I tried to listen to the physical disks activity and I hear nothing. My system is pretty silent (other than the usual fan noise).

The GUI also isn’t showing anything either. All the screens in the GUI are blank (except the shell, that I can launch through GUI).

I also read somewhere about /var/log/messages as it may have latest activities. However, the last message in this log is from 6-7 hours ago when I issued the zpool import command (with -f).

@essinghigh @Arwen
Is there any other thing I can try to attach these disks to the pool (or access the data on the disks)?

[Arwen]

That error message indicates that the pool is NOT imported. The pool got suspended due to un-correctable I/O errors. So it is not doing anything.

You can try with the “-F” option. Sometimes pool corruption occurs in the last write. Dropping that write makes the remaining data & pool good.

I don’t have the various pool import options memorized. I’d have to look them up in the manual pages, which you can do to. But, their are others to perform more extreme pool recoveries.

This type of problem is one reason why I suggest TrueNAS uses learn about ZFS. If you had off-lined the failing disk before shutting down, then that may have prevented your current problems.

ZFS is in someways overly complex. It was written for the Enterprise Data Center marker, for true server hardware. It CAN and DOES work perfectly fine for MANY, MANY uses, including NASes and media servers.

However, when trouble arrases, ZFS can sometimes require detailed command line trouble shooting. Where basic ZFS knowledge is helpful, even if reading the manual page is needed.

Other than what I at the top, I don’t have any immediate suggestions.

[maltafk]

Thank you @Arwen ,
This gives me a direction. Let me do bit reading and some more trials to see if this is resolved. In either case, I’ll update this post.

[maltafk]

@Arwen, @essinghigh
After reading documentation, and few other forums, seems I am making some progress. Will you be able to guide in a direction with current situation that I am facing.

After trying few commands (and meddling bit with the SATA cables/ports etc) CLI shows pool as well as disks as online:

root@HomeNAS[~]# zpool import
   pool: All4TBDisks
     id: 14329411509399635582
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        All4TBDisks                                     ONLINE
          raidz1-0                                      ONLINE
            gptid/7374a1f2-4796-11ed-980d-1027f526f244  ONLINE
            gptid/737ee777-4796-11ed-980d-1027f526f244  ONLINE
            gptid/7367fdfc-4796-11ed-980d-1027f526f244  ONLINE
            gptid/738c0045-4796-11ed-980d-1027f526f244  ONLINE

However, when I try to import the pool using any of the options (-F or -f or -FX or -R etc), I get different issues with the disks. Sometimes it says “Faulted Too many errors”, sometimes it says “UNAVAIL cannot open”. Here are output of few of the commands:

root@HomeNAS[~]# zpool import -Fn -R /mnt All4TBDisks
root@HomeNAS[~]# zpool import
   pool: All4TBDisks
     id: 14329411509399635582
  state: UNAVAIL
status: One or more devices are faulted.
 action: The pool cannot be imported due to damaged devices or data.
 config:

        All4TBDisks                                     UNAVAIL  insufficient replicas
          raidz1-0                                      UNAVAIL  insufficient replicas
            gptid/7374a1f2-4796-11ed-980d-1027f526f244  FAULTED  too many errors
            gptid/737ee777-4796-11ed-980d-1027f526f244  ONLINE
            gptid/7367fdfc-4796-11ed-980d-1027f526f244  UNAVAIL  cannot open
            gptid/738c0045-4796-11ed-980d-1027f526f244  ONLINE

root@HomeNAS[~]# zpool import -Fn -R /mnt All4TBDisks
root@HomeNAS[~]# zpool import
   pool: All4TBDisks
     id: 14329411509399635582
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
 config:

        All4TBDisks                                     UNAVAIL  insufficient replicas
          raidz1-0                                      UNAVAIL  insufficient replicas
            gptid/7374a1f2-4796-11ed-980d-1027f526f244  UNAVAIL  cannot open
            gptid/737ee777-4796-11ed-980d-1027f526f244  ONLINE
            gptid/7367fdfc-4796-11ed-980d-1027f526f244  UNAVAIL  cannot open
            gptid/738c0045-4796-11ed-980d-1027f526f244  ONLINE

I also found below command, and tried it with all the options like -F -f -FfX etc, but I keep getting same error (I/O error):

root@HomeNAS[~]# zpool import -f -o readonly=on All4TBDisks
cannot import 'All4TBDisks': I/O error
        Destroy and re-create the pool from
        a backup source.

zpool status shows only the boot pool.

root@HomeNAS[~]# zpool status -v
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:15 with 0 errors on Mon May 20 03:45:15 2024
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          ada2p2    ONLINE       0     0     0

errors: No known data errors

root@HomeNAS[~]# zpool import -F -R /mnt All4TBDisks
cannot import 'All4TBDisks': no such pool or dataset
        Destroy and re-create the pool from
        a backup source.


root@HomeNAS[~]# zpool import -Ff -R /mnt All4TBDisks
cannot import 'All4TBDisks': no such pool available


root@HomeNAS[~]# zpool import -FfX -R /mnt All4TBDisks
cannot import 'All4TBDisks': no such pool available

I am a bit lost again as I thought if everything comes online then I’ll be all set, but seems the behavior is becoming more and more random (with the errors related to the disks). Any further guidance will be highly appreciated.

[Arwen]

Sorry, I can’t make any further suggestions.

[maltafk]

@Arwen @essinghigh @Stux
Good news, I changed the PC, and moved all the disks (including TrueNAS OS disk) to the new PC. Everything started to work fine now.

The next (and priority) step for me is to now take a back up to avoid such stress in future if the disks/system fails again :-).

Although the old system (after installing windows on another disk to check what may be wrong with it) seems to be working fine so I am not sure what is causing the issue in the old system. But this is probably headache for another time :-).

Thank you for your time and ideas in trying to resolve my issue. I thought as the issue is resolve I should share it with you so you are aware of the solution that worked for and also have a closure of this issue :-).

Thank you and have a wonderful day.