Did my Installation get corrupt?

Hi,

I’ve recently move from an Xpenology setup to TrueNAS Scale.
I mainly use it as a backup / storage. I’m running the most recent version of SCALE

The OS is installed on a Kingston V300 SSD which I had laying around

As storage I had a 1TB Seagate drive which I use to play around with and a 2x2TB HDD in a mirror setup.

Since this week I’ve been running into multiple issues:

  1. SMB stopped working:
    The SMB share to the Dataset on the 2TB mirror doesn’t work anymore, nor the SMB Service won’t start at all giving an error. This has worked properly before as I was able to copy over all my data from my old NAS setup. When trying to start SMB service the following error pops up
[EFAULT] [Errno 28] No space left on device: '/var/db/system/samba4'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 208, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1526, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1457, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service.py", line 190, in start
    await service_object.start()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/cifs.py", line 16, in start
    if not await self.middleware.call("smb.configure_wait"):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1626, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1457, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/smb.py", line 457, in configure_wait
    await conf_job.wait(raise_error=True)
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 438, in wait
    raise CallError(self.error)
middlewared.service_exception.CallError: [EFAULT] [Errno 28] No space left on device: '/var/db/system/samba4'
  1. The app IMMICH crashed and wouldn’t start anymore.
    I used the 1TB to fool around with IMMICH and started pushing photo’s and video’s from two phones to this drive to free up space (and as a replacement of Google Photo’s).
    Earlier this week I opened the dashboard, and it showed the app crashed, and wouldn’t start anymore. Strangely enough it shows that the disk has 0 space left, which is odd, as the two phones are 128GB & 256GB, so should be more then enough space left on the 1TB.
    I thought it might help reinstalling IMMICH, but it wouldn’t allow me to install me any apps anymore on this disk, so I moved the app install location to the 2TB drives temporarily.
    Even though it was intended to just try-out stuff with IMMICH on this 1TB drive, I’d like to see if there is a way I can get the Immich database recovered (as there was already some hours invested in this, and some photos were only on left on Immich(I know very stupid)…

The 2TB drives can be completely erased, as that data still is on a backup NAS, so if I need to move things around, that is not a problem.

So my main goal is to see if I can recover the data that is on the 1TB drive, but I’ve got no idea how to access it at this moment.

Hopefully I can get some help from here.

Thanks in advance :slight_smile:

I suggest browsing the forums and reading a few threads. You can also do the tutorial. We need to get your forum trust level up so you have more posting rights, like images, etc.

TrueNAS-Bot
Type this in a new reply and send to bring up the tutorial, if you haven’t done it already.

@TrueNAS-Bot start tutorial

While you are doing that, can you type in System → Shell in the GUI?

zpool status -v

into the console window and paste the results back here using Preformatted text (Ctrl +e). It looks like </> on the toolbar above where you post.

1 Like

@SmallBarky thanks alot.

zpool status -v

results into:

zsh: command not found: zpool

try sudo zpool status -v

2 Likes

Add sudo in front of the command, like this:

sudo zpool status -v

Because the error message states that you ran out of disk space, please also include the following output:

sudo zpool list

2 Likes
  pool: Data
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
config:

        NAME                                      STATE     READ WRITE CKSUM
        Data                                      ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            a4ec3ab1-9a75-49b5-91db-0aad16c6a270  ONLINE       0     0     0
            f40f689a-f87d-41a2-9f86-c3cc302cbad9  ONLINE       0     0     0

errors: No known data errors

  pool: Rotzooi
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
config:

        NAME                                    STATE     READ WRITE CKSUM
        Rotzooi                                 ONLINE       0     0     0
          2c72fe69-8f7a-435e-874d-795e654d4695  ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:27 with 0 errors on Tue Nov 26 03:45:29 2024
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdb3      ONLINE       0     0     0

errors: No known data errors

and

NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Data       1.81T  1.31T   511G        -         -     0%    72%  1.00x    ONLINE  /mnt
Rotzooi     928G   899G  28.7G        -         -    30%    96%  1.00x    ONLINE  /mnt
boot-pool    27G  7.19G  19.8G        -         -    10%    26%  1.00x    ONLINE  -

There doesn’t seem to be anything wrong with the pool, looks like Rootzooi simply ran out of space. The system dataset is likely stored on that pool, which is why samba won’t start anymore.

Can you post the output of the following:

sudo zfs list -t all -r Rootzooi

It will show you how much space each dataset is taking (as well as list all,zvols,etc). You should also be able to use the Web-UI to check space usage.

NAME                                                                                                                        USED  AVAIL  REFER  MOUNTPOINT
Rotzooi                                                                                                                     899G     0B    96K  /mnt/Rotzooi
Rotzooi/.system                                                                                                            1.54G     0B  1.12G  legacy
Rotzooi/.system/configs-ae32c386e13840b2bf9c0083275e7941                                                                   2.08M     0B  2.08M  legacy
Rotzooi/.system/cores                                                                                                        96K     0B    96K  legacy
Rotzooi/.system/netdata-ae32c386e13840b2bf9c0083275e7941                                                                    426M     0B   426M  legacy
Rotzooi/.system/nfs                                                                                                         124K     0B   124K  legacy
Rotzooi/.system/samba4                                                                                                      424K     0B   204K  legacy
Rotzooi/.system/samba4@update--2024-10-31-06-18--SCALE-24.04.2.3                                                            120K      -   192K  -
Rotzooi/.system/samba4@update--2024-11-20-21-45--SCALE-24.10.0                                                              100K      -   196K  -
Rotzooi/AppConfig                                                                                                           140K     0B   140K  /mnt/Rotzooi/AppConfig
Rotzooi/ix-applications                                                                                                    4.19G     0B   256K  /mnt/Rotzooi/ix-applications
Rotzooi/ix-applications@ix-applications-backup-system-update--2024-10-31_06:18:39                                             0B      -   256K  -
Rotzooi/ix-applications/catalogs                                                                                           46.0M     0B  46.0M  /mnt/Rotzooi/ix-applications/catalogs
Rotzooi/ix-applications/default_volumes                                                                                      96K     0B    96K  /mnt/Rotzooi/ix-applications/default_volumes
Rotzooi/ix-applications/default_volumes@ix-applications-backup-system-update--2024-10-31_06:18:39                             0B      -    96K  -
Rotzooi/ix-applications/k3s                                                                                                4.14G     0B  4.14G  /mnt/Rotzooi/ix-applications/k3s
Rotzooi/ix-applications/k3s@ix-applications-backup-system-update--2024-10-31_06:18:39                                       184K      -  4.14G  -
Rotzooi/ix-applications/k3s/kubelet                                                                                         284K     0B   284K  legacy
Rotzooi/ix-applications/releases                                                                                           1.23M     0B   104K  /mnt/Rotzooi/ix-applications/releases
Rotzooi/ix-applications/releases@ix-applications-backup-system-update--2024-10-31_06:18:39                                    0B      -   104K  -
Rotzooi/ix-applications/releases/filebrowser                                                                                544K     0B   104K  /mnt/Rotzooi/ix-applications/releases/filebrowser
Rotzooi/ix-applications/releases/filebrowser@ix-applications-backup-system-update--2024-10-31_06:18:39                        0B      -   104K  -
Rotzooi/ix-applications/releases/filebrowser/charts                                                                         240K     0B   240K  /mnt/Rotzooi/ix-applications/releases/filebrowser/charts
Rotzooi/ix-applications/releases/filebrowser/charts@ix-applications-backup-system-update--2024-10-31_06:18:39                 0B      -   240K  -
Rotzooi/ix-applications/releases/filebrowser/volumes                                                                        200K     0B    96K  /mnt/Rotzooi/ix-applications/releases/filebrowser/volumes
Rotzooi/ix-applications/releases/filebrowser/volumes@ix-applications-backup-system-update--2024-10-31_06:18:39                0B      -    96K  -
Rotzooi/ix-applications/releases/filebrowser/volumes/ix_volumes                                                             104K     0B    96K  /mnt/Rotzooi/ix-applications/releases/filebrowser/volumes/ix_volumes
Rotzooi/ix-applications/releases/filebrowser/volumes/ix_volumes@ix-applications-backup-system-update--2024-10-31_06:18:39     0B      -    96K  -
Rotzooi/ix-applications/releases/filebrowser/volumes/ix_volumes/config                                                        8K     0B   108K  /mnt/Rotzooi/ix-applications/releases/filebrowser/volumes/ix_volumes/config
Rotzooi/ix-applications/releases/filebrowser/volumes/ix_volumes/storage_entry                                                 0B     0B    96K  /mnt/Rotzooi/ix-applications/releases/filebrowser/volumes/ix_volumes/storage_entry
Rotzooi/ix-applications/releases/immich                                                                                     612K     0B   104K  /mnt/Rotzooi/ix-applications/releases/immich
Rotzooi/ix-applications/releases/immich@ix-applications-backup-system-update--2024-10-31_06:18:39                             0B      -   104K  -
Rotzooi/ix-applications/releases/immich/charts                                                                              276K     0B   276K  /mnt/Rotzooi/ix-applications/releases/immich/charts
Rotzooi/ix-applications/releases/immich/charts@ix-applications-backup-system-update--2024-10-31_06:18:39                      0B      -   276K  -
Rotzooi/ix-applications/releases/immich/volumes                                                                             232K     0B    96K  /mnt/Rotzooi/ix-applications/releases/immich/volumes
Rotzooi/ix-applications/releases/immich/volumes@ix-applications-backup-system-update--2024-10-31_06:18:39                     0B      -    96K  -
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes                                                                  136K     0B   128K  /mnt/Rotzooi/ix-applications/releases/immich/volumes/ix_volumes
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes@ix-applications-backup-system-update--2024-10-31_06:18:39          0B      -   128K  -
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/library                                                            0B     0B    96K  /mnt/Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/library
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/pgBackup                                                           0B     0B    96K  /mnt/Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/pgBackup
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/pgData                                                             8K     0B   477M  /mnt/Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/pgData
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/profile                                                            0B     0B    96K  /mnt/Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/profile
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/thumbs                                                             0B     0B   923M  /mnt/Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/thumbs
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/uploads                                                            0B     0B  34.5G  /mnt/Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/uploads
Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/video                                                              0B     0B  95.0M  /mnt/Rotzooi/ix-applications/releases/immich/volumes/ix_volumes/video
Rotzooi/ix-apps                                                                                                             893G     0B   120K  /mnt/.ix-apps
Rotzooi/ix-apps/app_configs                                                                                                1.03M     0B  1.03M  /mnt/.ix-apps/app_configs
Rotzooi/ix-apps/app_mounts                                                                                                  888G     0B    96K  /mnt/.ix-apps/app_mounts
Rotzooi/ix-apps/app_mounts/filebrowser                                                                                      428K     0B   104K  /mnt/.ix-apps/app_mounts/filebrowser
Rotzooi/ix-apps/app_mounts/filebrowser@1.1.12                                                                                 0B      -   104K  -
Rotzooi/ix-apps/app_mounts/filebrowser/config                                                                               228K     0B   108K  /mnt/.ix-apps/app_mounts/filebrowser/config
Rotzooi/ix-apps/app_mounts/filebrowser/config@ix-applications-backup-system-update--2024-10-31_06:18:39                      60K      -   108K  -
Rotzooi/ix-apps/app_mounts/filebrowser/config@1.1.12                                                                         60K      -   108K  -
Rotzooi/ix-apps/app_mounts/filebrowser/storage_entry                                                                         96K     0B    96K  /mnt/.ix-apps/app_mounts/filebrowser/storage_entry
Rotzooi/ix-apps/app_mounts/filebrowser/storage_entry@ix-applications-backup-system-update--2024-10-31_06:18:39                0B      -    96K  -
Rotzooi/ix-apps/app_mounts/filebrowser/storage_entry@1.1.12                                                                   0B      -    96K  -
Rotzooi/ix-apps/app_mounts/immich                                                                                           888G     0B   112K  /mnt/.ix-apps/app_mounts/immich
Rotzooi/ix-apps/app_mounts/immich@1.6.24                                                                                      0B      -   112K  -
Rotzooi/ix-apps/app_mounts/immich/backups                                                                                  1.74G     0B  1.74G  /mnt/.ix-apps/app_mounts/immich/backups
Rotzooi/ix-apps/app_mounts/immich/backups@1.6.24                                                                             64K      -    96K  -
Rotzooi/ix-apps/app_mounts/immich/library                                                                                   208K     0B    96K  /mnt/.ix-apps/app_mounts/immich/library
Rotzooi/ix-apps/app_mounts/immich/library@ix-applications-backup-system-update--2024-10-31_06:18:39                          56K      -    96K  -
Rotzooi/ix-apps/app_mounts/immich/library@1.6.24                                                                             56K      -    96K  -
Rotzooi/ix-apps/app_mounts/immich/pgBackup                                                                                   96K     0B    96K  /mnt/.ix-apps/app_mounts/immich/pgBackup
Rotzooi/ix-apps/app_mounts/immich/pgBackup@ix-applications-backup-system-update--2024-10-31_06:18:39                          0B      -    96K  -
Rotzooi/ix-apps/app_mounts/immich/pgBackup@1.6.24                                                                             0B      -    96K  -
Rotzooi/ix-apps/app_mounts/immich/pgData                                                                                   1009M     0B   381M  /mnt/.ix-apps/app_mounts/immich/pgData
Rotzooi/ix-apps/app_mounts/immich/pgData@ix-applications-backup-system-update--2024-10-31_06:18:39                          388M      -   477M  -
Rotzooi/ix-apps/app_mounts/immich/pgData@1.6.24                                                                             169M      -   318M  -
Rotzooi/ix-apps/app_mounts/immich/profile                                                                                   592K     0B   472K  /mnt/.ix-apps/app_mounts/immich/profile
Rotzooi/ix-apps/app_mounts/immich/profile@ix-applications-backup-system-update--2024-10-31_06:18:39                          64K      -    96K  -
Rotzooi/ix-apps/app_mounts/immich/profile@1.6.24                                                                             56K      -   472K  -
Rotzooi/ix-apps/app_mounts/immich/thumbs                                                                                   6.92G     0B  6.85G  /mnt/.ix-apps/app_mounts/immich/thumbs
Rotzooi/ix-apps/app_mounts/immich/thumbs@ix-applications-backup-system-update--2024-10-31_06:18:39                         54.1M      -   923M  -
Rotzooi/ix-apps/app_mounts/immich/thumbs@1.6.24                                                                            14.6M      -  4.98G  -
Rotzooi/ix-apps/app_mounts/immich/uploads                                                                                   844G     0B   844G  /mnt/.ix-apps/app_mounts/immich/uploads
Rotzooi/ix-apps/app_mounts/immich/uploads@ix-applications-backup-system-update--2024-10-31_06:18:39                         596M      -  34.5G  -
Rotzooi/ix-apps/app_mounts/immich/uploads@1.6.24                                                                           11.6M      -  46.2G  -
Rotzooi/ix-apps/app_mounts/immich/video                                                                                    34.4G     0B  34.4G  /mnt/.ix-apps/app_mounts/immich/video
Rotzooi/ix-apps/app_mounts/immich/video@ix-applications-backup-system-update--2024-10-31_06:18:39                          36.5M      -  95.0M  -
Rotzooi/ix-apps/app_mounts/immich/video@1.6.24                                                                              416K      -   342M  -
Rotzooi/ix-apps/docker                                                                                                     5.06G     0B  5.06G  /mnt/.ix-apps/docker
Rotzooi/ix-apps/truenas_catalog                                                                                            6.77M     0B  6.77M  /mnt/.ix-apps/truenas_catalog

So it seems like Immich did do something crazy with the uploads causing it to fill up to 844G :sweat_smile:

How exactly I don’t understand (yet). But is there a way to remove anything that is not necessary (i.e. the Filebrowser & Docker folders can be removed).

Also I don’t understand why the system dataset is stored on this drive, and not on the SSD where the installation is done onto?

The boot disk is for the boot pool only.
The first data pool you create gets the system dataset unless you change it afterwards to another data pool

1 Like

Deleting Filebrowser isn’t going to give you much space. Docker is necessary to run immich.

It does look like you migrated from TrueNAS Core, is that correct?
You can delete the Rotzooi/ix-applications dataset, which will give you 4.19G of space. However you will loose the ability to go back to Core. ONLY DO THIS IF EVERYTHING MIGRATED CORRECTLY FROM CORE!
EDIT: Should be Scale 24.9.x instead of Core.

You are using app storage of type ixVolume which I am not familiar with. I always use Host Path mounts. You’re going to have the issue that deleting data from immich isn’t going to free up space because you have snapshots on the dataset. I don’t know how people are supposed to use ixVolume storage properly - I don’t see an option to delete restore points in the UI.

No I did a install from scratch.

So if I’d remove ix-applications, I might be able to restart SMB and also reinstall Immich and see how I can clean up some of the mess that that has made?

Yeah ignore what I said, it was the upgrade from pre EE to EE where the dataset changed from ix-applications to ix-apps.

Yes, I do think that deleting that dataset is enough. Make sure to delete the smaller dataset called ix-applications. Also if the immich data is important this may be a good time to create a backup.

Did you have Core installed initally? It looks like you went from Core to a version of Scale and then to the latest Scale version. A clean install only takes care of the boot device / boot pool.

Both your data related pools are pretty full. You want them below 80% capacity before the warnings start. At 90%, ZFS changes the way it saves the data and, as you saw, above that can cause crashes.

Correct. I installed just before EE got released and upgrade almost straight after that.

Removing IX-applications solved the SMB issue, I can now acces both my SMB Shares again. However the “Rotzooi” share doesn’t show any folders in Windows.

The immich data is going to be important at some stage, as it has all the pictures taken by the phone of my wife and I, however 95% of what was on the phones, still is on there, just a small part might be lost, so it’s not yet the end of the world. It’s more that I’d like to save the time invested in setting up Immich and make take some lessons learned of what happend.

Your share probably never showed any data before? You’re not going to see Immich related data. All immich related data is stored as /mnt/.ix-apps.

As @SmallBarky stated, your data pools are pretty full. You might look into acquiring more storage space.

You might be able to use the immich app now and maybe figure out where the unexpected space usage is coming from?

If you want to check out what data is in the uploads folder (844G) you can use the following command:
(DO NOT post the output, it might have private info)

sudo du -h /mnt/.ix-apps/app_mounts/immich/uploads
1 Like

Correct. The storage is full. My generic Data pool has some deduplication work to do.

The Immich stuff is probably also stuffed with duplicates (just an expectation as I previously mentioned, the source of what got put into Immich is somewhere around 400GB). In the process I removed Immich because it wasn’t working. So I have to find out a way how to re-install it and being able to connect it to the old data…

I’m getting the following output when trying to run that code:

du: cannot access '/mnt/.ix-apps/app_mounts/immich/uploads': No such file or directory

I think it is supposed to be

sudo du -h /mnt/Rotzooi/ix-apps/app_mounts/immich/uploads

Can you check how much free space you have? I just hope that the correct dataset got deleted.