Since upgrading to Fangtooth my apps catalog and app updates start failing after the system has been running for a few days. It seems like the .ix-apps folders get their permissions set to READ ONLY for some reason. Then the catalog and apps can not update. It will correct itself if I reboot the system. Anyone have any ideas??? I am now running Fangtooth (25.04.2.1)
How did you find out it gets set to read only?
The most common reason for the catalogue failing to update is due to a misconfigured or faulty network.
I am not saying that it canât be due to other reasons, just that network issues are the most common ones.
Either way, for anyone to be able to offer any real help you will need to provide more information about the error message (full error text), at the very least. Depending on the error more information will likely be requested.
Copy paste it and use the </> button in the text toolbar to preserve the formatting.
3 Likes
Most of the time when my apps fail its because netbird app Highjacked the network and the whole NAS canât connect to the internet anymore.
Go to the shell and try to ping a public server. Doesnât work? Network problem.
I didnât research the logs this time, but last time it happened I saw an log entry that indicated the catalog folder had been made read only.
My apps all continue to function and I see nothing that looks like a network issue. This time when it failed I found out because I had an app FAIL its update, so it appears more of the .ix-apps area is affected than just the catalog folder. All went back to working after the system was reboot as usual.
Same ERROR with catalog not updating again today. Here is one of the reasons why I think it is it is a Read Only issue courtesy of my AI helper:
root@truenas[/mnt/.ix-apps/truenas_catalog]# ls -ld
drwxr-xr-x 7 root root 17 Aug 23 14:38 .
root@truenas[/mnt/.ix-apps/truenas_catalog]#
I was ask to attempt to change the folder permissions:
chown -R middleware:middleware /mnt/.ix-apps/truenas_catalog
Which resulted in this error:
chmod: changing permissions of â/mnt/.ix-apps/truenas_catalog/cspell.config.yamlâ: Read-only file system
and many nore like it. . .
AI: gemma3:12b-it-qat
Okay, thatâs a crucial piece of information! The âRead-only file systemâ error indicates that the filesystem where /mnt/.ix-apps/truenas_catalog resides is mounted as read-only. This is preventing you from changing permissions on any files within the directory.
Understanding the Problem
The core issue is not the permissions themselves, but the fact that the filesystem is mounted read-only. This likely stems from a hardware or software configuration issue.
The chatbot is leading you astray.
Youâre not meant to be doing anything to the .ix-apps dataset manually. Thatâs why itâs hidden.
Can you show the error you get when you actually try to update the catalogue?
I have to ask, since you just tried to run chmod like that, have you been running manual commands before this started happening in a similar manner?
NO, I am not in the habit of going around the UI interface. I just tried the chmod command because I was pretty sure I was dealing with a Read Only situation from past logs I have encountered. This has been going on randomly since I first upgraded to 25.04. It started when I had Truenas running on bare metal hardware and continued after I moved the SATA drive pool to a VM on Promax v8. The Error always goes away for a while after the VM is restarted.
**** Canât show the error message NOW, I will have to post it when it fails next time. ****
Updated to 25.04.2.3 yesterday - Error still following my system around. Here is the less than helpful error message that get posted. . .
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/catalog/git_utils.py", line 34, in pull_clone_repository
clone_repository(repository_uri, destination, branch, depth)
File "/usr/lib/python3/dist-packages/middlewared/utils/git.py", line 25, in clone_repository
raise CallError(
middlewared.service_exception.CallError: [EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: fatal: destination path '/mnt/.ix-...
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 515, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 560, in __run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 93, in wrapped
result = await func(*args)
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/catalog/sync.py", line 27, in sync
await self.middleware.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1005, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 731, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/catalog/sync.py", line 55, in update_git_repository
return pull_clone_repository(repository, location, branch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/catalog/git_utils.py", line 36, in pull_clone_repository
raise CallError(f'Failed to clone {repository_uri!r} repository at {destination!r} destination: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: [EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: fatal: destination path '/mnt/.ix-...
This is interesting. It could be an issue with the file system or it could be a networking issue.
Can you ping (or curl) github.com from Truenas? Use the shell and ping guthub.com or curl github.com. To test the connection to GitHub.
I have not seen anything that makes it look like a network issue. Pinging github.com works just fine. What I do know about the issue is the catalog folder at: /mnt/.ix-apps/truenas_catalog turns up changed to Read Only when I get the failures. I had a failure a few days ago and discovered one of my apps also couldnât update it had a failure related to not being able to write to the apps folder, so Iâm guessing most of the .ix-apps data set is being left in a Read Only state when I am experiencing the failures. Rebooting Truenas always solves the issue for a few hours or a few days, I havenât been able to link the failures to anything the system is doing that triggers the failure.
Did you manage to resolve this? Have the exact same issue.
Version 25.04.2.4.
- Checked the permissions.
- Can resolve and ping other hosts, also external.
- tried every suggestion i could find in the last days.
- Even completely reinstalled from scratch (am in the progress of moving from Synology to TrueNAS)
Any directions would be greatly appreciated!
Are you running any form of adblock or security filtering software?
Do you use your ISPâs DNS or something else?
NO, I have not found a resolution to this problem. I also had it follow my main data pool thru a complete reinstall of TrueNAS 25. It does still goes away when the system is rebooted. My issue appeared after a working Nextcloud app FAILED to migrate from TrueNAS 23 to 24 and I have never been able to get Nextcloud to install since. That may or may not have caused the issue, but the timing goes with the issue.
This is NOT a DNS issue, the folder where the catalog is stored is preventing it from being updated because it is set to READ ONLY. After a reboot is will not be read only and do the update until it starts to fail again in a day or so.
I donât assume that the person I replied to has the same issue as you do.
It didnât occur to me the issue might have a different cause, but you are absolutely right it could not update because of a DNS issue. Thank you for staying in touch with this post. . .
1 Like
Thank you both for your helpful replies.
i managed to make it work by simply using another cable. I could resolve DNS etc, but there was no significant traffic possible. Getting a 100mb.bin started but then stalled. Now the refresh works.
3 Likes
I finally corrected this sync issue by DELETING the pool that contained the IX System Apps folder and recreating the pool from a backup. . .Worked fine until I upgraded to 25.10.1 - Goldeye last week now I have the exact same issue again. Wonât sync UNLESS I reboot the server. Using APPS on Truenas is a DISASTER waiting to happen. . .