Fresh 25.04 install got problems with apps

Apps and Virtualization
1

Hello there,

Here I am after burning OpenAI servers by trying to get ChatGPT help me to figure out why my apps won’t work at all on a brand new installation of Truenas 25.04.

A bit of context : I’m a Freenas Core user for a long time. Few years ago, I decided to migrate to Scale but without dealing with migration guide so I’ve lost lots of services in jails. Whatever, it’s my fault. I recently decided to get back to Truenas Core, according to ZFS version compatibility and I was very disappointed to see that Core developments were very limited compared to Scale. Moreover, if I wasn’t a K3S fan, I was very excited that Truenas make the choice to use docker instead and give us the ability to create our own apps from compose files. So yesterday, I finally decided to migrate my Core installation from the latest release (13.3 U1.1) to the latest Scale community stable release (25.04).

So here’s my problem. Truenas is a wonderful NAS and more system when everything run just fine but seems to became a pretty shit blackbox when it’s running into trouble that it cannot deal with alone. I freeshly installed (twice) Scale 25.04, set SSH access just to avoid navigating into system > shell to execute commands and then navigate to Apps to initialize the service on my pool (3 disks mirror that has been ZFS updated after the first installation as suggested by the GUI). And, badaboum.

docker.update freeze at 60%, no log or didn’t find any.
catalog.sync freeze at 5%, no log neither.

I let both tasks ran a complete night, disks led activity is never stopping but nothing seems to be done. Trying to unmount the pool from Apps but it is seen busy so I can’t and now I’m going to make a 3rd fresh install and trying to mount the Apps service on a brand new pool that I just initialized for testing purposes.

SO, I’m running out of ideas to deal with this shit and begin to regret my choice. From the moment I installed my first Freenas Core instance that ran perfectly for few years, I’m really thinking about migrating to Proxmox because “Stable” Truenas Scale are not even nightly release anymore.

I’m not a beginner in server administration because its my job but maybe I’ve missed something so why I’m requesting your help before running away from this.

2

Well well, I have a clue BUT the Truenas blackbox is giving me headache :slightly_smiling_face:

So after fresh install, I asked for Apps initialization on a single disk pool created by this version of Truenas, and it worked ! After inspecting zpools and how they’re mounted, the only difference I see is the acl type (NFSv4 on my fault pool vs posixacl on the one created directly through TN 25).

This could be an explanation since it already has been discussed here for volumes mounted in containers.

So I disconnected on drive from my pool, connected a USB one with the same size and create another pool aside from the one where all my data is. Without checking before replicating my data, I asked for apps initialization on that brand new pool and … It’s been created with NFSv4 ACL type. And docker is crashing again :joy:

Launching the pool wizard creation to see if I missed an advanced creation mode or something, well, it seems that just stiped disks (SSD) are created with Posix ACL but Mirror HDD are created with NFSv4 ACLs. Why ? How ? Hello black box.

I’m running out of idea and I cannot interrupted my useless replication task from the GUI so I’m just loising time before I’ll probably need to reinstall, AGAIN the system.

3

After migrating an old ZFS pool from FreeBSD-based TrueNAS to TrueNAS SCALE 25.04, I faced a lot of weird problems:

  • catalog.sync task freezing forever
  • docker.update also freezing
  • Unable to start Applications properly
  • No error logs, scrub clean, pool healthy

:magnifying_glass_tilted_left: Root Cause

Several old ZFS properties and ACL settings were incompatible with TrueNAS SCALE’s Docker requirements:

  • Wrong ACL types (NFSv4 vs POSIX)
  • Leftover ZFS flags (sync=always)
  • Inherited broken mount options
  • Bad compression ratios and weird NFS sharing settings

:broom: The Complete Fix

Assuming your pool is named Tank.

1. Check and Fix ACL Type

TrueNAS SCALE expects POSIX ACLs or clean POSIX permissions on system/application pools.

Check ACL type:

zfs get acltype Tank

If you see nfsv4, convert it to POSIX:

zfs set acltype=posix Tank

Make sure inherited files and folders are updated:

chmod -R 755 /mnt/Tank
chown -R root:root /mnt/Tank

(optional, adapt to your needs)

2. Clean ACL behavior (discard NFSv4 ACL entries)

Set ACL inheritance to discard bad ACLs:

zfs set aclmode=discard Tank
zfs set aclinherit=discard Tank

This ensures no exotic ACLs will propagate or break permission models.

3. Reset Important ZFS Properties

To avoid issues, also reset important properties:

zfs set atime=off Tank
zfs set compression=lz4 Tank
zfs set sharenfs=off Tank
zfs set sharesmb=off Tank
zfs set readonly=off Tank

Ensure devices, exec, and setuid are ON:

zfs set devices=on Tank
zfs set exec=on Tank
zfs set setuid=on Tank

4. Fix ZFS sync Property

This was critical!

Check current setting:

zfs get sync Tank

If it says always, it forces every write to wait for disk flush — extremely slow for Docker.

During my investigation process, I removed Log SSD disk from my pool to build a test pool aside in mirror with another SSD. Don’t know if its linked, but it could be.

Fix it:

zfs set sync=standard Tank

:white_check_mark: This improves performance and prevents system jobs like catalog.sync and docker.update from freezing.

5. Restart the System

Once the above fixes are applied:

systemctl restart middlewared

(or just reboot TrueNAS properly)

This reloads ZFS properties, ACLs, and middleware services.

:fire: Results

  • Now it works.

I hope it could help someone to not follow the “backup your data, destroy and recreate your pool” guys.

4

Think I might be encountering a similar problem and hopefully your solution works for me as well hopefully!

5

Hi chaps having the problem with catalog.sync that won’t go past 5% after a long time sitting doing nothing.

[EFAULT] Failed to clone (https:// github link) repository at ‘/mnt/.ix-apps/truenas_catalog’ destination: [EFAULT] Failed to clone (https:// github link) repository at ‘/mnt/.ix-apps/truenas_catalog’ destination: Cloning into ‘/mnt/.ix-apps/truenas_catalog’…

This is a total new build as I wanted to get away from Western Digital with it’s lack of apps so following great reviews on YouTube built a new server (I build computers Windows boxes) and installed TrueNAS, set up pools and users all no problem BUT first I can’t get a test email sent and second catalog.sync freezes so no apps! Getting very frustrated going through all the settings in the server and my Router and not seeing anything wrong.

I tried Ringoski’s shell commands which were promptly denied so no joy there.

Any help much appreciated.

PS apparently you can’t post links in these chats so replaced them with (https:// github link)

6

Catalogue sync issues are often due to time sync issues. If the time is too far out of sync GitHub refuses the connection.
Make sure your bios time is set to UTC and your local time is then set through the truenas gui

7

Thanks for the suggestion but all times are in sync and correlate with my PC’s clock.

I have found that it seems that TrueNAS cannot “get out” ie. it cannot send a test email and now I find that I cannot connect to MS One Drive either:

Error Name: EINVAL
Error Code: 22
Reason: HTTPSConnectionPool(host=‘graph.microsoft.com’, port=443): Max retries exceeded with url: /v1.0/me/drives (Caused by NewConnectionError(‘<urllib3.connection.HTTPSConnection object at 0x7f8068ced850>: Failed to establish a new connection: [Errno 101] Network is unreachable’))
Error Class: ConnectionError
Trace: Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/urllib3/connection.py”, line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/urllib3/util/connection.py”, line 96, in create_connection
raise err
File “/usr/lib/python3/dist-packages/urllib3/util/connection.py”, line 86, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 705, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 388, in _make_request
self._validate_conn(conn)
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 1050, in _validate_conn
conn.connect()
File “/usr/lib/python3/dist-packages/urllib3/connection.py”, line 358, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/urllib3/connection.py”, line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f8068ced850>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/requests/adapters.py”, line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 789, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/urllib3/util/retry.py”, line 594, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=‘graph.microsoft.com’, port=443): Max retries exceeded with url: /v1.0/me/drives (Caused by NewConnectionError(‘<urllib3.connection.HTTPSConnection object at 0x7f8068ced850>: Failed to establish a new connection: [Errno 101] Network is unreachable’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py”, line 361, in process_method_call
result = await method.call(app, id_, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py”, line 57, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 956, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 784, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 667, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 116, in wrapped
result = func(*args)
^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/rclone/remote/onedrive.py”, line 51, in list_drives
r = requests.get(
^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/requests/api.py”, line 73, in get
return request(“get”, url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/requests/api.py”, line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/requests/adapters.py”, line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host=‘graph.microsoft.com’, port=443): Max retries exceeded with url: /v1.0/me/drives (Caused by NewConnectionError(‘<urllib3.connection.HTTPSConnection object at 0x7f8068ced850>: Failed to establish a new connection: [Errno 101] Network is unreachable’))

I am wondering if there is something amiss between TrueNAS and my Router, but I have logged into the router and cannot see anything that looks different from the settings in TNAS. I am stumped.

8

are you using dhcp or a static ip?
If you’re using a static ip did you set the nameservers and default route (your routers ip)? those don’t stick if one switches from dhcp to static ip and have to be set manually

9

Hi Lars,

I am using dhcp and have checked the IP settings which are the same in the router as are set in TN. I did contemplate trying a static IP but have not had the router change any addresses yet.

Funny thing I found an email from the server this morning telling me of the catalog.sync failure so it can send emails even if it is 12 hrs later! Tried the apps cataogue again just now but no luck.

One other thing, if it means anything?, when I woke my PC this morning it had no internet and finally had to turn the router off and on again before it connected. My wife sitting next to me uses WiFi whilst I use CAT 6 connection to the server, her computer (ident to mine as I built them) was not affected.

Thanks for your help it is much appreciated to have someone with knowledge of networking that I do not trying to help.