Power Outage - Pool Offline - Can't Import Via GUI

Hi, I have a Dell Power Edge running Version: TrueNAS-13.0-U1.1

We had a power outage and after booting up our pool is showing as offline in the GUI.

When I run zpool Import via CLI it says it’s online (see zpool status response below too)

root@truenas[~]# zpool import
   pool: Photoshoots
     id: 13697090546972977673
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        Photoshoots                                   ONLINE
          gptid/7ffff484-23d3-11ed-aede-90b11c08801f  ONLINE
root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:02 with 0 errors on Thu Nov 21 03:45:02 2024

errors: No know data errors

When I go through the steps to import the pool the dropdown does not show any pools import.

I’ve searched around on the forums and found a thread on the old forum called reboot-and-pools-offline (102791) which was the closest thing to the problem I’m facing but OP had more of an idea than I do and I don’t want to go around doing stuff without understanding what I’m doing.

I’ve very limited experience with managing servers, I helped set this up 2 years ago with an IT tech (they’re currently unavailable) and after calling a handful of business this afternoon I’ve not had any joy finding anyone comfortable enough with TrueNAS to help.

Any help would be greatly appreciated.

Also, there is about 30-40TB of data on there, if I were to run an import that worked :pray: roughly how long would it take (so I can plan for staff)?

Thanks, Ben

1 Like

We will probably need your detailed hardware info

please run

zpool status -v

And post the results here in the Preformatted text, like you did above.

Hi, thanks for the reply. Here is the full status response.


root@truenas[~]# zpool status -v                                                
  pool: boot-pool                                                               
 state: ONLINE                                                                  
  scan: scrub repaired 0B in 00:00:02 with 0 errors on Thu Nov 21 03:45:02 2024 
config:                                                                         
                                                                                
        NAME        STATE     READ WRITE CKSUM                                  
        boot-pool   ONLINE       0     0     0                                  
        da0p2        ONLINE       0     0     0                                  
                                                                                
errors: No known data errors                                                    
   

There are 7x 8GB drives in there set up raid 5 (potentially RAID6).
Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
64GiB Ram

If there is anything else specifically needed I can provide when I’m back in the office in the morning or with instruction I can pull from CLI as I have remote access from home via GUI by connecting to a system on the same network)

I’m sure more experienced posters will reply with questions.

Do you have a backup elsewhere of the data on Photoshoots?
How are the 8 Gold drives attached to the system. Is there a HBA, what model?

Raid 5 isn’t a term for ZFS and TrueNAS. I think you mean you have a VDEV of 8 Gold 8GB drives in Raid-Z1.

To help with terminology of ZFS

Hi, thanks for the clarification.

Dell perc h710p 3.13
(taken from CLI dump response to cat /var/run/dmesg.boot)

We don’t have a direct back-up as such due to the nature of the files and the size of what is an ever moving target, the files aren’t quite ‘business critical’ in that sense but it would set us back a few weeks at best so really would like to ‘recover’ this.

Thanks again Ben

Not sure if you want to risk it (there shouldn’t be issues, but we’re already past that point).

I’m wondering what would happen if you export the pool with CLI & then if it’ll show up in GUI to import & then after importing it using GUI life is good again.

zpool export Photoshoots

Wondering if GUI is just behind & doing this would get things back into sync.

Edit: I tested in proxmox where I care a lot less about my VM data & it seems to have worked fine, but hard to say since I didn’t have the initial issues you are experiencing.

From the post on the old forum (see below) that seems to be what fixed the issue in that case, it seems like the pool was imported into the wrong directory somehow and an export/import corrected it but I don’t know if that’s definitely the case here.

1 Like

This will need to be checked as it may be providing RAID services and not allowing ZFS direct access to the disks. Someone with more familiarity with that card, I hope, will post. I don’t know how to check it with commands.

2 Likes

Could be possible, @jaminbenharrison - you flashed the thing to IT mode prior to deployment?

If I’m not mistaken, the following should let us know since has a LSISAS2208 controller:

sas2flash -list

One of the row should have something like:

Firmware Product ID : 0x2213 (IT)

The (IT) confirming it is in IT mode. However I’m still guessing that pool just needs to be exported using CLI & re-imported using GUI using:

zpool export Photoshoots

Hi, thanks again for the help.

I’ve just tried that and results …

image

…Odd - whats the output of?

lsblk -o NAME,SIZE,SERIAL,LABEL,FSTYPE

You can remove ‘serial’ if you want to keep your HDD serial#s hidden from us. I won’t be offended.

Edit: The following will show if the HBA is even showing (stole this from an old post from HoneyBadger)

lspci | grep LSI

I just get command not found with the lsblk

image

Oh crap, right - you’re on CORE.

gpart list

I want to make sure your hard drives are even being recognized.



root@truenas[~]# gpart list
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 248774615
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,64bc29d0-23ce-11ed-86ce-90b11c08801f,0x28,0x400)
   rawuuid: 64bc29d0-23ce-11ed-86ce-90b11c08801f
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 524288
   offset: 20480
   type: freebsd-boot
   index: 1
   end: 1063
   start: 40
2. Name: da0p2
   Mediasize: 110175977472 (103G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17180413952
   Mode: r1w1e1
   efimedia: HD(2,GPT,64c00450-23ce-11ed-86ce-90b11c08801f,0x2000428,0xcd38000)
   rawuuid: 64c00450-23ce-11ed-86ce-90b11c08801f
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 110175977472
   offset: 17180413952
   type: freebsd-zfs
   index: 2
   end: 248742951
   start: 33555496
3. Name: da0p3
   Mediasize: 17179869184 (16G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e1
   efimedia: HD(3,GPT,64bdf840-23ce-11ed-86ce-90b11c08801f,0x428,0x2000000)
   rawuuid: 64bdf840-23ce-11ed-86ce-90b11c08801f
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 17179869184
   offset: 544768
   type: freebsd-swap
   index: 3
   end: 33555495
   start: 1064
Consumers:
1. Name: da0
   Mediasize: 127372623872 (119G)
   Sectorsize: 512
   Mode: r2w2e4

Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 78134640599
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   efimedia: HD(1,GPT,7ff61cab-23d3-11ed-aede-90b11c08801f,0x80,0x400000)
   rawuuid: 7ff61cab-23d3-11ed-aede-90b11c08801f
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da1p2
   Mediasize: 40002788438016 (36T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   efimedia: HD(2,GPT,7ffff484-23d3-11ed-aede-90b11c08801f,0x400080,0x1230efff58)
   rawuuid: 7ffff484-23d3-11ed-aede-90b11c08801f
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 40002788438016
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 78134640599
   start: 4194432





Also shows like this in the GUI

Uhh - I’m guessing you don’t have a single ~36TB disk that makes up your pool.

I’m guessing that SmallBarky is right (also given the output of the sas2flash command you provided earlier).

I think something went a bit wrong during your power outage & that you HBA is acting in IR (raid controller) instead of IT mode currently. Instead of passing through the HDDs to TrueNAS to control, it is presenting them as a single device.

If I’m looking at the right thing from Dell, you should be able to set disks for straight passthrough (Converting Physical Disk To Non-RAID For PERC H310) to the system as per:

…Otherwise you might have to flash the thing into IT mode. After that, hope that disks are detected again individually & can be properly important by TrueNAS; otherwise smarter people than me will need to help.

Fleshmauler, I think you looked at wrong Dell card. Dell perc h710p 3.13 is what OP posted

Should be relevant, no? H710P is listed, but I could always be mistaken. I still suspect that the H710P isn’t in IT like you posted earlier.

Only other time I’ve heard of drives being clumped as a single device other than HBA being in RAID is when they are routed through USB enclosure.

Thanks again for the help with this. Really appreciate it.

So rather than them showing as individual discs it’s showing as 1 single disc.

Is there a way to check if it was like this before too? I feel like (for what it’s worth) it has always been like that, as in I don’t ever remember seeing a list of discs there… Is there a way to tell?

What would happen if we changed it and it was like this from the start, can we change it back?

Also if we were to try the export/import of the pool as it is, would it fail and fail in a way that causes fatal issues?

I didn’t click the link and view pdf, sorry