TrueNAS stuck with some services after a disk has been replaced

Not able to do anything, I’m stuck with those services that are unable to start

I wanna put a print screen but I can’t…
on Jobs page, those services won’t start;
smb.configure
pool.dataset.sync_db_keys
directoryservices.setup
pool.import_on_boot

How bad did you break it? Did you have a pool go bad after a disk died, replaced the disk, resilvered the pool, now it’s saying it’s healthy again? Or is it still sick?

Right now it’s super sick! My pool have 4 HDDs. On the GUI it says :
CRITICAL
Pool FreeNAS state is DEGRADED: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.
The following devices are not healthy:

So I don’t know what to do

@Denys_Laflamme
Welcome to the TrueNAS Forums.

You have really provided very little usable information to help you. My take away is you have a ZFS pool issue, and it may be a failed drive.

My advice, use the link in my signature or go to the resources page and use the Drive Troubleshooting Flowcharts. Do not try to do too many things at once, it will just bite you in the rear. Do one thing at a time, meaning address one problem, fix it, move on to the next problem. Hopefully you only have one problem, a failed drive. Maybe you are using an SMR drive and replacing it fixes all your problems at once.

If that doesn’t help, refer to my other link called Joes Rules and post the data asked for so someone can assist you better. You would need to provide your system hardware setup, the version of TrueNAS you are using, and some details on how this problem started. Try to remember one very important thing, do not assume we know what is going on with your system. As a group, we are very good at helping people however we need to know specific things to provide you a good response.

Best of luck to you.

Joe is right, which is why I was trying to get more info. You live and die by the health of your pools and barely mentioned them. Now we know you have 4 drives with one dead or dying so that drive has been offlined. Still need to know more.

What does the pool consist of? Is it raidz1 with all 4 drives working together in a single pool? Dip into your Storage icon and get some detail there, preferably a screenshot, blur the parts you don’t want to show us but we need the drive count, and the health condition of each drive.

It’s not panic time yet, if things are running, they’re just in danger. The faster we can point at the fault, the faster we can fix it.

Tanks to all.
Yes afrosheen, I’ve 4 HDD SATA drive. And I know that they’re connected :

So the sda is the system, sdb is the new one no yet attached to the pool.
sdc is the oldest 3To that goes with sdd 4To and sde 4To.
The sdc, sdd and sde works fine until I put the new one sdb.

I can’t run any zpool command it give no prompt back.

The boot process is super long, about 16 minutes and it tells that ix-etc.service can’t start:

Also in the GUI, impossible to get any screen with Storage:

fdisk also give me this:

I’m running the OS Version:TrueNAS-SCALE-24.04.2.5

So I’m still have no clue what do to. I’m able to run SMART test on every drive…

Sdb looks like a rip and replace, that drive doesn’t even have a partition. Maybe you jumped the gun and yanked it then threw this one in. Waiting to see your smart results.

That’s it’s not what I expected, my prompt is back after a second…

@Denys_Laflamme Make sure you track the drives by serial number. If you do a lot of rebooting, it is quite possible that the drives grab a different Device ID. It doesn’t happen often but the fact is, it can happen.

What is the output of zpool status -v? You do not need to wait for the SMART test to complete.

Also, are you able to grab the text and paste it into the forum vice screen captures? Why do I ask for this? It is searchable text. I can’t search an image. Use the </> in the menu line above.

1 Like

Thanks joeschmuk, I’ll use </> next time.
But as I epexted, the zpool status -v show nothing, the prompt never get back on zpool command (like I wrote before).

Please humor me, I’m not there so I will ask questions like I’m stupid. It is an attempt to remove assumptions so do not take offence.

Please list your hardware specs.

What are you using to access the TrueNAS computer, directly connected keyboard or IPMI or SSH?

Are you logged in as root?

How long after logging in as root does it take to see a command prompt?

Do you have a copy of your TrueNAS configuration files? When you “upgrade” truenas, it asks you to save the keys. This is important to have a copy, you may really need to use it.

Above all, do not rush the process of fixing this, it could lead to confusion between you and the forum members trying to help you.

What I want to do is wait on those answers, but then have you power down the system, disconnect the data drives in your pool, power back up. How does the system react? It should easily boot up. It could take time as it tried to mount the pool but wait it out.

Thx joeschmuck, I’m not offended, and you’re not stupid either… I appreciate all helps…
I log root on the GUI shell or on the KVM attached to the NAS… Command prompt not taking much time to appear.
For the TrueNAS config files, I guess I’ve those somewhere on my drive, can’t tell where exactly… At the default location I guess…
I tried to boot with no HDD attached and it was really fast, but no pool off course!

I’ve a stupid question, but I know that /dev/sd* is related to the place on the board each drive are connected. I remember, after getting trouble to load the pool, tried to change data cable on the board. Does it affect the way that the pool should be discover ?
But in my situation, it seems that the pool won’t be discover in the way I connect my SATA drives… :frowning:

New things now!!!
I use this command:

systemctl disable ix-zfs.service

so after a reboot, a quick reboot, I’m able to see this on Storage:

But there’s no way for me to import the pool…

root@truenas[~]# zpool import -f -F   
  pool: FreeNAS
    id: 8909234627546120194
 state: DEGRADED
status: One or more devices are offlined.
action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
config:

        FreeNAS                                   DEGRADED
          raidz1-0                                DEGRADED
            7afb11f8-610c-465f-98f3-ea6b47a01063  OFFLINE
            4bc11e54-f900-428c-ad19-a9d2433d1401  ONLINE
            c97fab7f-8a17-4c56-8e98-201619a832a7  ONLINE
            0d391935-b9b7-4d39-a6ca-04d54781f7ce  ONLINE

So not able to do anything with this;

zpool import -f -F FreeNAS

Any advice guys ?

After searching, I see this post really interesting and close to my case :

So I run this command :

echo 1 > /sys/module/zfs/parameters/zfs_recover
zpool import FreeNAS -R /mnt -o readonly=on

And now my pool is back again in read-only but able to list all the files!
So now, how to make it back to normal ?

So you can read your degraded pool after importing it in recovery mode, and since it’s raidz1-0 I think that means you have parity across the three surviving drives in the pool. Now you need to add that new drive to the pool and have it resilver, which will heal your pool and away you go.

But, I don’t have the answer, I replaced a failing drive by offlining it, pulling it, then putting a fresh drive in and selecting something.

Hoping we get someone to come along and help answer this, you are so close to the finish line now. :crossed_fingers:

Now you need to add that new drive to the pool and have it resilver, which will heal your pool and away you go.

Yeah that’s what I think, but if I don’t import it in read-only mode, I can’t add my new disk on it as it said read-only :frowning: So I try to scrub it but no success…

Be patient now, @HoneyBadger can’t be in every thread. We’re just missing a little more advice before you solve this, don’t tinker further until someone else checks in. And yes, you can’t do this in readonly mode because you wanna write something out to that new drive when it’s joined so the pool has to be r/w.