Disk failure in raidz1, pool unavailable and remaining disks is exported

TrueNAS General
, ,
1

HELP!!!

Just encountered a problem where my TrueNAS crashed with Kernel Panic message on screen. It was Panic: zfs: adding existent segment to range tree…

Could not get my server to start again when power off/power on, it just stopped at this stage. Eventually I tried to disconnect all harddrives and recoonected them one by one to see if there was something wrong with them. Finally I was able to get it up and running again but without one disk. That disk was part of a raid5 pool so I guess truenas should be able to handle that loss but at least it hasnt done so automatically.

My problem is that the other 2 drives in the raid 5 pool is in exported state and the pool isnt available.

image
image1752×519 28 KB

Anyone who know why they doesnt work without the missing drive or what I should do to get them up and running again???

OS Version:TrueNAS-SCALE-24.10.2.4

Product:System Product Name

Model:Intel(R) Core™ i5-6600K CPU @ 3.50GHz

Memory:47 GiB

2

No “raid5” with ZFS… You may mean “raidz1” (single parity array), but that would require at least three drives and your screenshot shows two pairs of drives in two different pools.

Please describe your NAS better. Bare metal? All drives attached directly to the motherboard? How many pools, and how many drives in each?
What’s the output of
sudo zpool status and
sudo zpool import if a pool is missing?
Please post the result as formatted text (use the </> button).

I’d suggest to power down, check and reseat all data and power cables, and power on again.

3

Yes, that is correct. I mean raidz1.

The bootup disk i a ssd m.2 disk (465,76 GiB) connected directly to the motherboard

I had 3 disks (3,64 TiB) connected directly to the motherboard and a pool named tank that consisted of theese 3 disks in a raidz1 setup.
I also had 2 other disks (7,28 TiB) connected through a sata PCI-E controller and configured them into a pool named tank2 which was mirrored

The system crashed and I had to disconnected 1 of the 3 drives in the raidz1 tank just too get the system up and running again, that why its not showing

zpool status:
pool: boot-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using ‘zpool upgrade’. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:02:16 with 0 errors on Mon Dec 8 03:47:17 2025
config:

    NAME         STATE     READ WRITE CKSUM
    boot-pool    ONLINE       0     0     0
      nvme0n1p3  ONLINE       0     0     0

errors: No known data errors

pool: tank2
state: ONLINE
scan: scrub repaired 0B in 01:39:15 with 0 errors on Sat Dec 6 02:39:17 2025
config:

    NAME                                      STATE     READ WRITE CKSUM
    tank2                                     ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        f47a689e-c981-4e0d-8406-d50650d43590  ONLINE       0     0     0
        728d5c1e-f6c8-4082-8b2d-415463d947c5  ONLINE       0     0     0

errors: No known data errors

zpool import:
pool: tank
id: 3992104908906065782
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the ‘-f’ flag.
see: Message ID: ZFS-8000-5E — OpenZFS documentation
config:

    tank                                      FAULTED  corrupted data
      raidz1-0                                DEGRADED
        973fb752-4187-4c92-85d3-4bcfab92208c  UNAVAIL
        59869b78-fc45-4ba5-8f90-5e4b6dd6297a  ONLINE
        a188ba46-37f0-4548-90da-ea6de758e3ef  ONLINE
4

Ouch! Do you have a backup?
Can you plug this drive back? Is it still working?

This is where you need to bring the third drive online, or lose data…
Import with just -f is likely to fail with some informative error message (post as formatted text please, </> button).
sudo zpool import -f -R /mnt tank

Beyond that, I’d like to know about the condition of the missing drive, and have @HoneyBadger or @Arwen double-check any potentially destructive command.

5

Heres the output from zpool import

admin@truenasscale[~]$ sudo zpool import -f -R /mnt tank[sudo] password for admin:cannot import ‘tank’: I/O errorRecovery is possible, but will result in some data loss.Returning the pool to its state as of Wed Dec 10 16:13:49 2025should correct the problem.  Approximately 18 minutes of datamust be discarded, irreversibly.  After rewind, severalpersistent user-data errors will remain.  Recovery can be attemptedby executing ‘zpool import -F tank’.  A scrub of the poolis strongly recommended after recovery.

Does that mean that it is possible to do a recovery with the suggested command?

6

The problem with the disk that I have disconnected is that the following crash occurs if I connect it. Dont know where to find the error messages in the logs. And dont know how to use the disk again. The strange thing is that why the system didnt handle that 1 disk seems to have crashed when I had it arranged as raidz1?

IMG_0900
IMG_09001632×1224 221 KB

7

This is where the logs end in syslog:

Dec  9 23:38:02 truenasscale kernel: eeepc-wmi eeepc-wmi: Detected ASUSWMI, use DCTSDec  9 23:38:02 truenasscale kernel: input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input8Dec  9 23:38:02 truenasscale kernel: snd_hda_intel 0000:01:00.1: Disabling MSIDec  9 23:38:02 truenasscale kernel: snd_hda_intel 0000:01:00.1: Handle vga_switcheroo audio clientDec  9 23:38:02 truenasscale systemd[1]: Listening on systemd-rfkill.socket - Load/Save RF Kill Switch Status /dev/rfkill Watch.Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC887-VD: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:lineDec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x11/0x0Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    inputs:Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:      Line=0x1aDec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9Dec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10Dec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11Dec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12Dec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13Dec  9 23:38:02 truenasscale kernel: RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timerDec  9 23:38:02 truenasscale kernel: RAPL PMU: hw unit of domain pp0-core 2^-14 JoulesDec  9 23:38:02 truenasscale kernel: RAPL PMU: hw unit of domain package 2^-14 JoulesDec  9 23:38:02 truenasscale kernel: RAPL PMU: hw unit of domain dram 2^-14 JoulesDec  9 23:38:02 truenasscale kernel: input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input14Dec  9 23:38:02 truenasscale kernel: input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input15Dec  9 23:38:02 truenasscale kernel: input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input16Dec  9 23:38:02 truenasscale kernel: input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input17Dec  9 23:38:02 truenasscale systemd[1]: Reached target sound.target - Sound Card.Dec  9 23:38:02 truenasscale kernel: cryptd: max_cpu_qlen set to 1000Dec  9 23:38:02 truenasscale kernel: AVX2 version of gcm_enc/dec engaged.Dec  9 23:38:02 truenasscale kernel: AES CTR mode by8 optimization enabledDec  9 23:38:03 truenasscale systemd[1]: Mounting boot-grub.mount - /boot/grub…Dec  9 23:38:03 truenasscale systemd[1]: var-lib-machines.mount - Virtual Machine and Container Storage (Compatibility) was skipped because of an unmet condition check (ConditionPathExists=/var/lib/machines.raw).Dec  9 23:38:03 truenasscale systemd[1]: Reached target machines.target - Containers.Dec  9 23:38:03 truenasscale systemd[1]: Mounted boot-grub.mount - /boot/grub.Dec  9 23:38:03 truenasscale kernel: intel_rapl_common: Found RAPL domain packageDec  9 23:38:03 truenasscale kernel: intel_rapl_common: Found RAPL domain coreDec  9 23:38:03 truenasscale kernel: intel_rapl_common: Found RAPL domain dramDec  9 23:38:03 truenasscale systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization.Dec  9 23:38:03 truenasscale systemd[1]: zfs-import-cache.service - Import ZFS pools by cache file was skipped because of an unmet condition check (ConditionFileNotEmpty=/etc/zfs/zpool.cache).Dec  9 23:38:03 truenasscale systemd[1]: Reached target zfs-import.target - ZFS pool import target.Dec  9 23:38:03 truenasscale systemd[1]: Starting zfs-mount.service - Mount ZFS filesystems…Dec  9 23:38:03 truenasscale systemd[1]: Starting zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev…Dec  9 23:38:03 truenasscale zvol_wait[902]: No zvols found, nothing to do.Dec  9 23:38:03 truenasscale systemd[1]: Finished zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev.Dec  9 23:38:03 truenasscale systemd[1]: Reached target zfs-volumes.target - ZFS volumes are ready.Dec  9 23:38:03 truenasscale systemd[1]: Finished zfs-mount.service - Mount ZFS filesystems.Dec  9 23:38:12 truenasscale systemd[1]: Finished ix-wait-on-disks.service - Wait on Disk Enumeration.Dec  9 23:38:12 truenasscale systemd[1]: Starting middlewared.service - TrueNAS Middleware…Dec  9 23:38:22 truenasscale middlewared[937]: /usr/lib/python3/dist-packages/pydantic/json_schema.py:2158: PydanticJsonSchemaWarning: Default value <object object at 0x7f2c470f8d40> is not JSON serializable; excluding default from JSON schema [non-serializable-default]Dec  9 23:38:22 truenasscale middlewared[937]:   warnings.warn(message, PydanticJsonSchemaWarning)

and in messages:

Dec  9 23:38:02 truenasscale kernel: iTCO_wdt iTCO_wdt: Found a Intel PCH TCO device (Version=4, TCOBASE=0x0400)Dec  9 23:38:02 truenasscale kernel: iTCO_wdt iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)Dec  9 23:38:02 truenasscale kernel: ee1004 0-0051: 512 byte EE1004-compliant SPD EEPROM, read-onlyDec  9 23:38:02 truenasscale kernel: ee1004 0-0052: 512 byte EE1004-compliant SPD EEPROM, read-onlyDec  9 23:38:02 truenasscale kernel: ee1004 0-0053: 512 byte EE1004-compliant SPD EEPROM, read-onlyDec  9 23:38:02 truenasscale kernel: input: PC Speaker as /devices/platform/pcspkr/input/input7Dec  9 23:38:02 truenasscale kernel: asus_wmi: ASUS WMI generic driver loadedDec  9 23:38:02 truenasscale kernel: asus_wmi: Initialization: 0x0Dec  9 23:38:02 truenasscale kernel: asus_wmi: BIOS WMI version: 0.9Dec  9 23:38:02 truenasscale kernel: asus_wmi: SFUN value: 0x0Dec  9 23:38:02 truenasscale kernel: eeepc-wmi eeepc-wmi: Detected ASUSWMI, use DCTSDec  9 23:38:02 truenasscale kernel: input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input8Dec  9 23:38:02 truenasscale kernel: snd_hda_intel 0000:01:00.1: Disabling MSIDec  9 23:38:02 truenasscale kernel: snd_hda_intel 0000:01:00.1: Handle vga_switcheroo audio clientDec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC887-VD: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:lineDec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    dig-out=0x11/0x0Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:    inputs:Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18Dec  9 23:38:02 truenasscale kernel: snd_hda_codec_realtek hdaudioC0D0:      Line=0x1aDec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input9Dec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10Dec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11Dec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12Dec  9 23:38:02 truenasscale kernel: input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13Dec  9 23:38:02 truenasscale kernel: RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timerDec  9 23:38:02 truenasscale kernel: RAPL PMU: hw unit of domain pp0-core 2^-14 JoulesDec  9 23:38:02 truenasscale kernel: RAPL PMU: hw unit of domain package 2^-14 JoulesDec  9 23:38:02 truenasscale kernel: RAPL PMU: hw unit of domain dram 2^-14 JoulesDec  9 23:38:02 truenasscale kernel: input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input14Dec  9 23:38:02 truenasscale kernel: input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input15Dec  9 23:38:02 truenasscale kernel: input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input16Dec  9 23:38:02 truenasscale kernel: input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input17Dec  9 23:38:02 truenasscale kernel: cryptd: max_cpu_qlen set to 1000Dec  9 23:38:02 truenasscale kernel: AVX2 version of gcm_enc/dec engaged.Dec  9 23:38:02 truenasscale kernel: AES CTR mode by8 optimization enabledDec  9 23:38:03 truenasscale kernel: intel_rapl_common: Found RAPL domain packageDec  9 23:38:03 truenasscale kernel: intel_rapl_common: Found RAPL domain coreDec  9 23:38:03 truenasscale kernel: intel_rapl_common: Found RAPL domain dram

and in middlewared.log

[2025/12/09 23:38:17] (INFO) middlewared.init():934 - Starting TrueNAS-SCALE-24.10.2.4 middleware[2025/12/09 23:38:26] (DEBUG) middlewared.setup():41 - Setting timezone to ‘Europe/Stockholm’[2025/12/09 23:38:31] (DEBUG) middlewared.generate_keys():199 - Generating SSH host keys[2025/12/09 23:38:32] (DEBUG) middlewared.setup():25 - Certificate setup for System complete[2025/12/09 23:38:32] (INFO) RDMAService._get_link_choices():67 - Fetching RDMA link netdev choices[2025/12/09 23:38:33] (DEBUG) middlewared.__plugins_setup():1105 - All plugins loaded[2025/12/09 23:38:33] (DEBUG) middlewared.__initialize():2114 - Accepting connections[2025/12/09 23:38:33] (INFO) DiskService.log_disk_info():76 - Found disks: {‘sda’: {‘serial’: ‘ZW60921F’, ‘lunid’: ‘5000c500e5b32580’}, ‘sdb’: {‘serial’: ‘ZW60DSNS’, ‘lunid’: ‘5000c500e635ae5c’}, ‘sdc’: {‘serial’: ‘ZW60A81V’, ‘lunid’: ‘5000c500e5bb4e6c’}, ‘sdd’: {‘serial’: ‘WWZ8Y0FZ’, ‘lunid’: ‘5000c500faeaf501’}, ‘sde’: {‘serial’: ‘WWZ8YCJR’, ‘lunid’: ‘5000c500faeb14a4’}, ‘nvme0n1’: {‘serial’: ‘PNF05230035710600049’, ‘lunid’: ‘nvme.1e4b-504e463035323330303335373130363030303439-504e592043533232333020353030474220535344-00000001’}}[2025/12/09 23:38:34] (DEBUG) PoolService.import_on_boot():464 - Creating ‘/data/zfs’ (if it doesnt already exist)[2025/12/09 23:38:34] (DEBUG) PoolService.import_on_boot():471 - Creating ‘/data/zfs/zpool.cache’ (if it doesnt already exist)[2025/12/09 23:38:34] (DEBUG) PoolService.import_on_boot_impl():309 - Importing ‘tank’ with guid: ‘3992104908906065782’

cant find the crash message anywhere, but maybe im looking in the wrong log files?

And by the way, how to format the text with </> so its just not a long line?

8

Normally, pasting the text and then selecting and clicking on </> should give the expected result.
Alternatively, type two lines of three backquotes and paste between these lines
```
PASTE HERE
```

This looks like one hard drive died in such a spectacular way that it took down the whole system. If the resulting panic caused damage to other drives, that’s more than a raidz1 can cope with.
Assuming that the drive is gone for good (and a data recovery service too expensive), and that losing 18 minutes is acceptable, the command to try would be
sudo zpool import -fFn tank
‘-n’ means dry run, returning without any error message would indicate potential success.
Then “the real thing” would be
sudo zpool import -fF -R /mnt tank
with TrueNAS alternate mount point. Backup important data out of there and scrub.

1 Like
9

unfortunately that did not work.

the dryrun worked without errors but when trying to force, the system crashed with the same error as before

10

Now to get the system up and running again I had to disconnect another disk of thoose 3, otherwise I couldnt get the system started…

1 Like
11

Do you have another system at hand where you could install TrueNAS and try these drives?
Maybe it’s the current motherboard/CPU/PSU. Otherwise, I’m afraid there’s not much hope.