Pool is now offline and drives appear as unassigned disks :/

Suddenly my pool has gone. I’m 100% sure that disks are fine since I made 2 days ago S.M.A.R.T. Test and it passed. Also I can’t import is for some reason. How do I make it work again? Sorry for beeing dumb :confused:
Screenshots: imgur /a/n89FhgD

Please give us your detailed hardware info.

Whats the output of zpool status -v
And

zpool import

Is this a bare metal install or a VM ?
How are your discs connected ?

( helping to get there ) The link has more images than shown here in the link image

…but I guess this is the important one

…yes, that will need answering @user1

1 Like
  1. Please run the following commands from a shell and copy & paste the results here:
  • sudo zpool status -v
  • DONE: sudo zpool import
  • lsblk -bo NAME,MODEL,PTTYPE,TYPE,START,SIZE,ROTA,PARTTYPENAME,PARTUUID
  • lspci
  • sas2flash -list
  • sas3flash -list
  1. The above might give this information, but we need to know the exact model of your drives and what the SATA controller is (motherboard or precise HBA model). And as previously suggested, we also need to know whether this is a native TrueNAS install or a VM under e.g. Proxmox.

Thx.

2 Likes

copy and paste the text

1 Like
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:09 with 0 errors on Mon Oct 21 03:45:10 2024
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors
root@truenas[~]# lsblk -bo NAME,MODEL,PTTYPE,TYPE,START,SIZE,ROTA,PARTTYPENAME,PARTUUID
NAME     MODEL                PTTYPE TYPE     START          SIZE ROTA PARTTYPENAME             PARTUUID
sda      KINGSTON SHSS37A240G gpt    disk            240057409536    0                          
├─sda1                        gpt    part      4096       1048576    0 BIOS boot                c50e473d-b22b-4a00-816d-83457544144e
├─sda2                        gpt    part      6144     536870912    0 EFI System               a75fa65e-bc76-4af5-bb4a-a69d2a30edfe
├─sda3                        gpt    part  34609152  222337506816    0 Solaris /usr & Apple ZFS e7b7a4f7-a23a-42ab-ac67-238c91a6365e
└─sda4                        gpt    part   1054720   17179869184    0 Linux swap               50d3b2b6-2d37-40db-b760-01f85ee7e353
  └─sda4                             crypt            17179869184    0                          
sdb      WDC WD60EFAX-68JH4N1 gpt    disk           6001175126016    1                          
├─sdb1                        gpt    part       128    2147483648    1 FreeBSD swap             5315203c-4f0b-11ee-ba55-e0d55e81a24e
└─sdb2                        gpt    part   4194432 5999027556352    1 FreeBSD ZFS              532bc192-4f0b-11ee-ba55-e0d55e81a24e
sdc      WDC WD60EFAX-68JH4N1 gpt    disk           6001175126016    1                          
├─sdc1                        gpt    part       128    2147483648    1 FreeBSD swap             530a3bb5-4f0b-11ee-ba55-e0d55e81a24e
└─sdc2                        gpt    part   4194432 5999027556352    1 FreeBSD ZFS              531f62da-4f0b-11ee-ba55-e0d55e81a24e
root@truenas[~]# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:14.0 USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]
00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)
00:1b.2 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #19 (rev f0)
00:1b.3 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #20 (rev f0)
00:1b.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0)
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
00:1c.2 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #3 (rev f0)
00:1c.3 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #4 (rev f0)
00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0)
00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation 200 Series PCH LPC Controller (Z270)
00:1f.2 Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
00:1f.4 SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
root@truenas[~]# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18) 
Copyright (c) 2008-2014 LSI Corporation. All rights reserved 

        No LSI SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.
root@truenas[~]# sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02) 
Copyright 2008-2017 Avago Technologies. All rights reserved.

        No Avago SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.
root@truenas[~]# 

I use two WD RED HDD WD60EFAX Motherboard - Gigabyte Z270P-D3 and this is native TrueNAS install

1 Like

if you copy 3 ` in a row it will make a box like so and press enter

where you can dump all the text you get

…then you press enter again and close that box with another 3 ` consecutively.

1 Like

WD60EFAX are SMR drives and so completely and utterly unsuitable for any use as ZFS redundant drives.

It looks like you made the pool with FreeNAS/TrueNAS Core and have subsequently upgraded. Is this the case? If so, when did you upgrade from CORE to SCALE?

Damn I didn’t knew that they are unsuitable

I upgraded from Core to Scale 3 months ago. They weren’t importing via import button but after I reinstall to Core exported config and imported to Scale, after that they imported fine. Actually texting this I might install Core again and try to import them but idk if that’s the best solution

It is your system, so feel free to take whatever actions you decide, but if you want your data back then my advice is to NOT to take any actions until a couple of knowledgeable people here reach consensus on how you might be able to recover your data.

1 Like

Well WD snuck SMR drives into their NAS range without telling anyone and sold them as suitable for NAS use for some time before they were called out over it, so perhaps they are to blame.

In essence, SMR drives do some fancy stuff to squeeze in more data, and this fancy stuff has a BIG write performance penalty. To counter that they divide the drive into two areas, mainly SMR butg also a small-ish CMR area which they use as a staging area - and when the drive is idle, they de-stage the data to the SMR area.

But in write intensive environments, the CMR cache fills up and then writes slow to a crawl. And in a resilver, this is exactly what happens, and the resilver takes days or even weeks instead of hours, during which time all drives are stressed and so more likely that another drive fails.

Assuming that we can help you bring the pool back online (perhaps read-only) do you have somewhere else you can write your data to?

1 Like

I don’t know if it is the best solution, but this is what I would do, preferably installing CORE to another drive so I could just swap TrueNAS version by swapping the boot drive (or with a trip to the BIOS boot menu). At least, it should be safe.
And SMR drives may be bad, but they are no explanation for the pool disappearing…

What’s the output of
smartctl -a /dev/sdb
smartctl -a /dev/sdc

We actually need the output of;
sudo zpool import
Without the pool name, it will give us the pool layout.

At times someone thinks they have made a Mirror pool or other redundant pool, and did not. Loss of 1 disk in any non-redundant pool will mean loss of whole pool.

Good point, but the output is in post #3 (kindly released from Imgur link by @argumentum ) and it is a mirror.

Screenshot 2024-10-22 212554

He did. Shows I/O Error :face_with_peeking_eye:

Sorry, I missed that.

If the SMART info says the disks look good, then the pool has corruption. (Can we blame Western Digital and their SMR firmware? I really want too!)

Their are options using ZFS write transaction roll backs, but they are a last resort. The exact command options are in some ways up to the user and how much they want to attempt. The “zpool-import” manual page has the details… I have not bothered to memorize them as I’ve not needed to use them personally.

Definitely we can blame WD for them being unsuitable for ZFS and also blame WD for mis-selling them.

But probably not for the pool corruption.

I don’t t

The pool corruption is unsettling.

Back to that old theory, bug, hdd issue, or memory error :wink:

Did anything happen before this? A crash? A blackout? Etc

I’d suggest suggest smart long tests on both drives and furnishing results.

The error is I/O Error.

Were these drives the type of SMR that gives up when the going gets tough with an IDNF errro?

That’d be an i/o error.

Thanks @Arwen

Nothing. I just booted and got this.