Fresh 25.04 install got problems with apps

Apps and Virtualization
1

Hello there,

Here I am after burning OpenAI servers by trying to get ChatGPT help me to figure out why my apps won’t work at all on a brand new installation of Truenas 25.04.

A bit of context : I’m a Freenas Core user for a long time. Few years ago, I decided to migrate to Scale but without dealing with migration guide so I’ve lost lots of services in jails. Whatever, it’s my fault. I recently decided to get back to Truenas Core, according to ZFS version compatibility and I was very disappointed to see that Core developments were very limited compared to Scale. Moreover, if I wasn’t a K3S fan, I was very excited that Truenas make the choice to use docker instead and give us the ability to create our own apps from compose files. So yesterday, I finally decided to migrate my Core installation from the latest release (13.3 U1.1) to the latest Scale community stable release (25.04).

So here’s my problem. Truenas is a wonderful NAS and more system when everything run just fine but seems to became a pretty shit blackbox when it’s running into trouble that it cannot deal with alone. I freeshly installed (twice) Scale 25.04, set SSH access just to avoid navigating into system > shell to execute commands and then navigate to Apps to initialize the service on my pool (3 disks mirror that has been ZFS updated after the first installation as suggested by the GUI). And, badaboum.

docker.update freeze at 60%, no log or didn’t find any.
catalog.sync freeze at 5%, no log neither.

I let both tasks ran a complete night, disks led activity is never stopping but nothing seems to be done. Trying to unmount the pool from Apps but it is seen busy so I can’t and now I’m going to make a 3rd fresh install and trying to mount the Apps service on a brand new pool that I just initialized for testing purposes.

SO, I’m running out of ideas to deal with this shit and begin to regret my choice. From the moment I installed my first Freenas Core instance that ran perfectly for few years, I’m really thinking about migrating to Proxmox because “Stable” Truenas Scale are not even nightly release anymore.

I’m not a beginner in server administration because its my job but maybe I’ve missed something so why I’m requesting your help before running away from this.

2

Well well, I have a clue BUT the Truenas blackbox is giving me headache :slightly_smiling_face:

So after fresh install, I asked for Apps initialization on a single disk pool created by this version of Truenas, and it worked ! After inspecting zpools and how they’re mounted, the only difference I see is the acl type (NFSv4 on my fault pool vs posixacl on the one created directly through TN 25).

This could be an explanation since it already has been discussed here for volumes mounted in containers.

So I disconnected on drive from my pool, connected a USB one with the same size and create another pool aside from the one where all my data is. Without checking before replicating my data, I asked for apps initialization on that brand new pool and … It’s been created with NFSv4 ACL type. And docker is crashing again :joy:

Launching the pool wizard creation to see if I missed an advanced creation mode or something, well, it seems that just stiped disks (SSD) are created with Posix ACL but Mirror HDD are created with NFSv4 ACLs. Why ? How ? Hello black box.

I’m running out of idea and I cannot interrupted my useless replication task from the GUI so I’m just loising time before I’ll probably need to reinstall, AGAIN the system.

3

After migrating an old ZFS pool from FreeBSD-based TrueNAS to TrueNAS SCALE 25.04, I faced a lot of weird problems:

  • catalog.sync task freezing forever
  • docker.update also freezing
  • Unable to start Applications properly
  • No error logs, scrub clean, pool healthy

:mag: Root Cause

Several old ZFS properties and ACL settings were incompatible with TrueNAS SCALE’s Docker requirements:

  • Wrong ACL types (NFSv4 vs POSIX)
  • Leftover ZFS flags (sync=always)
  • Inherited broken mount options
  • Bad compression ratios and weird NFS sharing settings

:broom: The Complete Fix

Assuming your pool is named Tank.

1. Check and Fix ACL Type

TrueNAS SCALE expects POSIX ACLs or clean POSIX permissions on system/application pools.

Check ACL type:

zfs get acltype Tank

If you see nfsv4, convert it to POSIX:

zfs set acltype=posix Tank

Make sure inherited files and folders are updated:

chmod -R 755 /mnt/Tank
chown -R root:root /mnt/Tank

(optional, adapt to your needs)

2. Clean ACL behavior (discard NFSv4 ACL entries)

Set ACL inheritance to discard bad ACLs:

zfs set aclmode=discard Tank
zfs set aclinherit=discard Tank

This ensures no exotic ACLs will propagate or break permission models.

3. Reset Important ZFS Properties

To avoid issues, also reset important properties:

zfs set atime=off Tank
zfs set compression=lz4 Tank
zfs set sharenfs=off Tank
zfs set sharesmb=off Tank
zfs set readonly=off Tank

Ensure devices, exec, and setuid are ON:

zfs set devices=on Tank
zfs set exec=on Tank
zfs set setuid=on Tank

4. Fix ZFS sync Property

This was critical!

Check current setting:

zfs get sync Tank

If it says always, it forces every write to wait for disk flush — extremely slow for Docker.

During my investigation process, I removed Log SSD disk from my pool to build a test pool aside in mirror with another SSD. Don’t know if its linked, but it could be.

Fix it:

zfs set sync=standard Tank

:white_check_mark: This improves performance and prevents system jobs like catalog.sync and docker.update from freezing.

5. Restart the System

Once the above fixes are applied:

systemctl restart middlewared

(or just reboot TrueNAS properly)

This reloads ZFS properties, ACLs, and middleware services.

:fire: Results

  • Now it works.

I hope it could help someone to not follow the “backup your data, destroy and recreate your pool” guys.

1 Like