Dataset missing after a notification of a snapshot failure due to out of disk space

Apps and Virtualization
1

Hi,

I’m running TrueNAS Scale and I had several apps installed and a VM with Home Assistant.

Today i noticed that some entities of the HA were not working. Went to TrueNAS machine and saw the following notification:
Critical Snapshot Task For Dataset “Apps/vm” failed: cannot create snapshot ‘Apps/vm@auto-2026-04-26_00-00’: out of space..

I removed some old snapshots and rebooted TrueNAS. When it came online the Dataset Apps was gone with the Apps and the VM.

Doing ls -l /mnt have this:

drwxr-xr-x 2 root root 2 Aug 20 2025 Apps
drwxr-xr-x 6 root root 6 Feb 11 17:50 DataStore

Apps is empty.

Doing zfs list it doesn’t show

On Storage Apps it says Data VDEVs Offline

I have 4 2.7T disks.

From lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT i get:

NAME     SIZE FSTYPE     TYPE MOUNTPOINT
sda      2.7T            disk
└─sda1   2.7T zfs_member part
sdb      2.7T            disk
└─sdb1   2.7T zfs_member part
sdc      2.7T            disk
└─sdc1   2.7T zfs_member part
sdd    223.6G            disk
├─sdd1     1M            part
├─sdd2   512M vfat       part
└─sdd3 223.1G zfs_member part
sde      2.7T            disk
└─sde1   2.7T zfs_member part
sr0     1024M            rom

Any help?

Thanks in advance

2

Is Apps/vm a dataset or zvol?

This will help us understand the usage and dataset hierarchy.

zfs list -r -t fs,vol -o space Apps

Pool history can help too.

zpool history Apps | grep -e destroy -e rename | grep -v @

Did you create a checkpoint?

zpool status Apps | grep checkpoint

EDIT: This is not relevant since the pool is offline.

I missed this in your post.

Your pool is offline.

Do you see it listed as importable?

zpool import
3

Thanks.No, it says that there is no pool to import.

4

What about the commands I blurred?

Do they show anything?

Are the drives even listed in your lsblk command? We don’t know what the Apps pool is made of.

5

Oh sorry, didn’t noticed since I saw on the phone.

Is Apps/vm a dataset or zvol? I’m almost sure it was a dataset. Is there a way to check that?

zfs list -r -t fs,vol -o space Apps:

cannot open ‘Apps’: dataset does not exist

zpool history Apps | grep -e destroy -e rename | grep -v @:

cannot open ‘Apps’: no such pool

I had these in Data Protection

image
image1170×1084 67.9 KB

6

Noticed also now that in /var/log/messages I have the following logs:
Apr 30 09:30:52 truenas kernel: WARNING: Pool ‘Apps’ has encountered an uncorrectable I/O failure and has been suspended.
Apr 30 09:31:23 truenas kernel: zio pool=Apps vdev=/dev/disk/by-partuuid/82b0844b-98ec-4d66-beb4-a5c1825d0065 error=5 type=2 offset=70635380736 size=94208 flags=1572992

This last one I have a bunch of them today

7

It appears your drives might have failed or disconnected. How are they connected? Are they listed in lsblk?

8

yes, it shows on the original post.

The server wasn’t mover for a long time. I will detach and attach the drives anyway.

9

your original post doesn’t show what pool the drives belong to from the LSBLK command. We can only go off what you explicitly post on the forums.

1 Like
10

Not sure what you are asking. You mean the physical drives?
My server is an old DELL server with 4 x 2.7 TB and are configured as mirror.

This is what lsblk shows:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 2.7T 0 disk
└─sda1 8:1 0 2.7T 0 part
sdb 8:16 0 2.7T 0 disk
└─sdb1 8:17 0 2.7T 0 part
sdc 8:32 0 2.7T 0 disk
└─sdc1 8:33 0 2.7T 0 part
sdd 8:48 0 2.7T 0 disk
└─sdd1 8:49 0 2.7T 0 part
sde 8:64 0 223.6G 0 disk
├─sde1 8:65 0 1M 0 part
├─sde2 8:66 0 512M 0 part
└─sde3 8:67 0 223.1G 0 part
sr0 11:0 1 1024M 0 rom

11

You have two pools. Which drives go to which pool?

12

There is no information that tells us that those drives belong to any specific pool or pools. We can’t see your entire system or what you are looking at. We don’t know what configuration your Apps pool is, nor the Data pool.

You have to tell us what we are expecting to see versus what is showing. A complete detailed listing of your hardware, OS version and the pool layout would help. You can also run the following command and posting the results back as it should show the current state of some of your system. Use Preformatted Text mode. You used it for a few command results already. (</>) or Ctrl+e on the toolbar

sudo zpool import
sudo zpool status -v
1 Like
13

Ok, I think I saw the problem. A few months ago i moved to an SSD 256gb the Apps and VM to do some testing with Frigate and forgot about it…

I mounted the SSD in another PC and it’s completely full. Is there a way to mount it to remove something, like snapshots?

14

How come those SSDs do not show up in the lsblk output?

16

Ok, so I have been looking and try to remember what I had implemented. Basically at the time I saw some documentation and guides on how to setup TrueNAS Scale.

TrueNAS is installed in one SSD (256GB). I remember that did it so that all the other 4 drives, which are drives for servers, stay dedicated to data.

The DataStore dataset is on the other 4x drives as mirror.

Now I also remember that I had another pool called Apps and inside I had a dataset app_configs. And that pool was selected the Apps. After installing more and more apps I realised the I should put everything together but my time now is short for development…

This latest pool was created on another separated SSD (128GB), and this one is the one that is corrupted..

Now from these logs:

1- Critical Snapshot Task For Dataset “Apps/vm” failed: cannot create snapshot ‘Apps/vm@auto-2026-04-26_00-00’: out of space..

2 - truenas kernel: WARNING: Pool ‘Apps’ has encountered an uncorrectable I/O failure and has been suspended.

3 - truenas kernel: zio pool=Apps vdev=/dev/disk/by-partuuid/82b0844b-98ec-4d66-beb4-a5c1825d0065 error=5 type=2 offset=70635380736 size=94208 flags=1572992

It seems that the disk went out of space or failed and pool got suspended. This was before I noticed that something was wrong in the HA and before I rebooted TrueNAS.

I noticed now that this 128 SSD is corrupted, this one I can’t mount anywhere. On Mac OSX it doesn’t show nothing regarding the device, Windows disk shows as undefined, on TrueNAS only if I install it through a USB drive I can see something, but it says 0 bytes and in dmesg I see the following:

<5>[ 10.764479] sd 8:0:0:0: [sde] Unit Not Ready
<6>[ 10.764612] sd 8:0:0:0: [sde] Sense Key : Hardware Error [current]
<6>[ 10.764617] sd 8:0:0:0: [sde] ASC=0x44 <>ASCQ=0x81
<3>[ 190.774827] sd 8:0:0:0: [sde] tag#0 timing out command, waited 180s
<3>[ 370.790929] sd 8:0:0:0: [sde] tag#0 timing out command, waited 180s
<3>[ 550.806904] sd 8:0:0:0: [sde] tag#0 timing out command, waited 180s
<6>[ 550.807094] sd 8:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
<6>[ 550.807105] sd 8:0:0:0: [sde] Sense Key : Hardware Error [current]
<6>[ 550.807112] sd 8:0:0:0: [sde] ASC=0x44 <>ASCQ=0x81
<3>[ 730.822880] sd 8:0:0:0: [sde] tag#0 timing out command, waited 180s
<5>[ 730.823074] sd 8:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B)
<5>[ 730.823206] sd 8:0:0:0: [sde] 0-byte physical blocks
<3>[ 910.826855] sd 8:0:0:0: [sde] tag#0 timing out command, waited 180s
<4>[ 910.827053] sd 8:0:0:0: [sde] Test WP failed, assume Write Enabled
<3>[ 1090.842829] sd 8:0:0:0: [sde] tag#0 timing out command, waited 180s
<3>[ 1090.843026] sd 8:0:0:0: [sde] Asking for cache data failed
<4>[ 1090.843156] sd 8:0:0:0: [sde] Assuming drive cache: write through
<5>[ 1090.843766] sd 8:0:0:0: [sde] Attached SCSI disk
<3>[ 1120.906851] sd 8:0:0:0: [sde] tag#0 timing out command, waited 30s
<3>[ 1138.706851] sd 8:0:0:0: [sde] tag#0 timing out command, waited 30s
<3>[ 1209.514893] sd 8:0:0:0: [sde] tag#0 timing out command, waited 60s
<3>[ 1209.583020] sd 8:0:0:0: [sde] tag#0 timing out command, waited 60s
<3>[ 1269.546802] sd 8:0:0:0: [sde] tag#0 timing out command, waited 60s
<3>[ 1269.586218] sd 8:0:0:0: [sde] tag#0 timing out command, waited 60s
<3>[ 1313.553952] sd 8:0:0:0: [sde] tag#0 timing out command, waited 15s
<3>[ 1328.570339] sd 8:0:0:0: [sde] tag#0 timing out command, waited 15s
<3>[ 1373.565528] sd 8:0:0:0: [sde] tag#0 timing out command, waited 15s

Already checked for short circuits in the SSD board but everything seems to be fine and actually it doesn’t heat on a specific location of the board.

Do you agree that the data got lost for good?

From the HA, i have a backup with more than a year. I lose some data and changes but it saves me quite some time if I would implement from scratch…

The only App that I want to be careful is Immich to see if I can recover everything and not mess up.

In the DataStore I have the Immich folder with the following datasets:

DataStore/Photos/Immich

DataStore/Photos/Immich/backup

DataStore/Photos/Immich/library

DataStore/Photos/Immich/pgData

DataStore/Photos/Immich/upload

DataStore/Photos/Immich/video

What is the best way to recover? Install the App with the data in a new location, and move/copy everything to the new location and then restore the latest backup? Or just use the same location and restore?

Sorry for the late reply but the weekend was busy.

Thanks

17

I hate to say it, but if the SSD is dead, then your entire pool is toast. Data recovery software would need drives that can be scanned in the first place.

That doesn’t appear to be a failure of unreadable cells, but a firmware failure of the SSD.

18

Yes, but all the dataset I selected for Immich is the good one. I can see all the fotos, videos and backups there. Actually I already accessed them and the photos are fine, only organised by Immich way. So the worst case I download all the photos and upload them again. The thing is that it’s a lot of GB’s.

I guess I would probably do that, just in case something goes wrong, I’m not losing the photos. But would be nice to recover from the backup so that our phones keep the sync and not having to deal with duplicates.

Thanks so much for yours and @SmallBarky inputs!

19

That’s your storage pool.

The “Apps” pool is what would contain the Immich metadata and software. Not as important, but the database will need to be recreated again.