HELP... I do get crazy .. importing pool hangs with zio_deadman ... hung I/O

TrueNAS General
, ,
1

Today I did a system reboot of my truenas core cos the vms and middlewared hung.

Now during restart I do see zio_deadman(): zio_wait waiting for hung I/O to pool …

how can I start and get more infos ? the boot pool is at a different ssd

I exchanged data cables and controller already.
Is there a way to boot without importing the pool and afterwards check disk by disk ?

2

So in the meantime I was able to do a readonly import after switching off all disks and powering them step by step during truenas is running.
zpool import -o readonly=on
worked fine pool was imported healty…
But a restarted system with readwrite was a fault again.

3

No idea in the wild to bring the pool back and import RW … ?

4

It would be helpful to post:

  • Hardware details
  • Disk make & model, (and if they are SMR or CMR)
  • How the disks are wired up to the server
  • TrueNAS version
  • Output of zpool import without attempting to import pool
5

4x ST8000NM0055-1RM. These are CMR drives.
Currently they are connected to an LSI SAS2008 in IT Mode via sas/sata breakout cable. Same issue at a SATA Controller (internal or 10Gtek PCIe SATA X4 Card 6 Port)
Mainboard ASUS PRIME X370-PRO wit 32 GB ECC RAM
AMD Ryzen 5 PRO 4650G
Truenas core
TrueNAS-13.0-U6.2

6 
% zpool import
   pool: zpool3-1
     id: 973410930289283071
  state: ONLINE
status: Some supported features are not enabled on the pool.
	(Note that they may be intentionally disabled if the
	'compatibility' property is set.)
 action: The pool can be imported using its name or numeric identifier, though
	some features will not be available without an explicit 'zpool upgrade'.
 config:

	zpool3-1          ONLINE
	  raidz1-0        ONLINE
	    gpt/ZA1JJNSN  ONLINE
	    gpt/ZA1JL7QY  ONLINE
	    gpt/ZA1JQNSJ  ONLINE
	    gpt/ZA1JRB05  ONLINE
7

Try zpool import -F -n zpool3-1 and share the output.

8

runs now for 2,5 days and reports … nothing but still runs

zpool events -vf show also nothing new on a second terminal

9

Not a great signal.

You can always remove the -F and -n to try importing for real the pool.

10

I would say I stop here my investigation and recreate the the Pool out of my backups.
But its a pity that there is no deeper analysis possible.
Is there a need to get the drives running like for the smartctl longterm test ?