Script to Fix App Catalog Sync Errors

I setup a Terramaster F8 Plus with TrueNAS Scale 25.04.1, and was experiencing app catalog sync errors that I couldn’t fix via the web interface. Others solved the issue by setting the bios time or manually unsetting/setting the app pool, but those didn’t work well for me. I think the main issue is how TrueNAS handles the network bridge I have setup.

To resolve the app catalog sync issues permanently, I created a script that manually syncs the app catalog via a cron job, checks that certain conditions are in place, and handles common errors automatically. After a lot of work, and some help from ChatGPT, I came up with the following to fix the app sync errors.

Problem:
If you’re encountering this recurring error on TrueNAS SCALE:

[EFAULT] Failed to clone ‘GitHub - truenas/apps’ repository at ‘/mnt/.ix-apps/truenas_catalog’ destination…

…it often prevents you from syncing the app catalog, especially during scheduled tasks. In some cases, the issue persists until the apps pool is unset and reset manually in the UI.

Root Cause (Suspected):

  • Having a network bridge setup appears to trigger the issue for me.
  • Catalog sync errors seem to be caused by the /mnt/.ix-apps/truenas_catalog dataset being in a weird mount state or the underlying middleware process getting jammed.
  • Manually unsetting and resetting the Apps pool in the UI always resolves the issue — suggesting an underlying bug in how the system handles app catalog mounts during sync.

Solution:
The solution is a script that checks for the catalog error, and if it exists:

  1. Unsets the app pool.
  2. Waits a few seconds.
  3. Resets the same app pool.
  4. Syncs the catalog.
  5. Logs actions and timestamps to /var/log/fix_ix_apps.log.

Directions:

To create the script, you will need to use shell. Shell can be accessed either from the TrueNAS web UI (Shell tab) or by connecting through SSH (e.g., with Termux, JuiceSSH, or PuTTY).

  1. Open Nano to Create the Script
sudo nano /root/fix_ix_apps.sh
  1. Paste This Script

Once nano is open, add the script text below.

#!/bin/bash

# ========================
# TrueNAS App Catalog Fix Script
# ========================
# ✅ Checks for TRUENAS catalog sync errors
# ✅ If present, unsets and resets the Kubernetes app pool
# ✅ Triggers catalog refresh
# ✅ Verifies success and logs outcome
# ✅ Includes .ix-apps sanity check
# ✅ Retries GitHub check up to 30 minutes before proceeding
# ========================

# Set log path
LOG="/var/log/fix_ix_apps.log"

# Timestamp
echo "=== $(date) ===" >> "$LOG"
echo "🔧 Starting TrueNAS app catalog fix script..." >> "$LOG"

# Step 0: Check if GitHub is reachable (retry for up to 30 minutes)
for attempt in {1..6}; do
  if curl -Is https://github.com --max-time 10 >/dev/null 2>&1; then
    echo "🌐 GitHub is reachable." >> "$LOG"
    break
  else
    echo "⏳ GitHub unreachable. Attempt $attempt/6. Retrying in 5 minutes..." >> "$LOG"
    sleep 300
  fi

  if [ "$attempt" -eq 6 ]; then
    echo "❌ GitHub still unreachable after 30 minutes. Exiting." >> "$LOG"
    exit 1
  fi
done

# Step 1: Check for catalog sync error
ERROR_MSG=$(midclt call alert.list | grep -i 'Failed to sync TRUENAS catalog')
if [ -z "$ERROR_MSG" ]; then
  echo "✅ No catalog sync error detected. No action needed." >> "$LOG"
  exit 0
else
  echo "❗ Catalog sync error detected. Proceeding with fix..." >> "$LOG"
fi

# Step 2: Unset the app pool
echo "🔧 Unsetting Kubernetes app pool..." >> "$LOG"
midclt call kubernetes.update '{ "pool": null }' >> "$LOG" 2>&1
sleep 3

# Step 3: Reset the app pool
echo "🔧 Re-setting Kubernetes app pool to 'storage'..." >> "$LOG"
midclt call kubernetes.update '{ "pool": "storage" }' >> "$LOG" 2>&1
sleep 5

# Step 4: Trigger catalog refresh
echo "🔄 Triggering app catalog sync..." >> "$LOG"
midclt call catalog.sync >> "$LOG" 2>&1
sleep 5

# Step 5: Check for persistent error
POST_ERROR=$(midclt call alert.list | grep -i 'Failed to sync TRUENAS catalog')
if [ -z "$POST_ERROR" ]; then
  echo "✅ Catalog sync succeeded after reset." >> "$LOG"
else
  echo "❌ Catalog sync error still present after reset." >> "$LOG"
fi

# Step 6: Sanity check - ensure .ix-apps is writable
if mount | grep -q '/mnt/.ix-apps'; then
  if [ ! -w /mnt/.ix-apps ]; then
    echo "❌ .ix-apps is mounted but not writable!" >> "$LOG"
    echo "📋 ZFS pool status:" >> "$LOG"
    zpool status >> "$LOG"
  else
    echo "✅ .ix-apps is writable." >> "$LOG"
  fi
else
  echo "⚠️ .ix-apps is not mounted!" >> "$LOG"
fi

echo "🏁 Script complete." >> "$LOG"
  1. Save and Exit in Nano
  • Press Ctrl + O to write the file
  • Press Enter to confirm the filename
  • Press Ctrl + X to exit nano
  1. (Optional) Schedule the Script to Run Automatically

To make this permanent, you will need to setup a cron job. Otherwise, you will need to run the script manually. The command below will make the script run automatically.

chmod +x /root/fix_ix_apps.sh
  1. Check the Script’s Log

To verify the script is running as intended, check the log file. ChatGPT is great at interpreting the log. The command below will show you the log details.

tail -n 50 /var/log/fix_ix_apps.log

Final Thoughts:

This script is working well for me. If you run into issues with the script, feel free to post your issue or modified script here.

1 Like

Excellent, been putting off making something like this for myself as I have weird issues with my bridge only on boot specifically.

Thoughts on doing a check on if github is reachable prior to the un/reset? Otherwise not much point imo

That would be a good addition.

If github were unreachable, it could wait a while and checking again.

1 Like

I updated the script so it now checks if github is reachable.

#!/bin/bash

# ========================
# TrueNAS App Catalog Fix Script
# ========================
# ✅ Checks for TRUENAS catalog sync errors
# ✅ If present, unsets and resets the Kubernetes app pool
# ✅ Triggers catalog refresh
# ✅ Verifies success and logs outcome
# ✅ Includes .ix-apps sanity check
# ✅ Retries GitHub check up to 30 minutes before proceeding
# ========================

# Set log path
LOG="/var/log/fix_ix_apps.log"

# Timestamp
echo "=== $(date) ===" >> "$LOG"
echo "🔧 Starting TrueNAS app catalog fix script..." >> "$LOG"

# Step 0: Check if GitHub is reachable (retry for up to 30 minutes)
for attempt in {1..6}; do
  if curl -Is https://github.com --max-time 10 >/dev/null 2>&1; then
    echo "🌐 GitHub is reachable." >> "$LOG"
    break
  else
    echo "⏳ GitHub unreachable. Attempt $attempt/6. Retrying in 5 minutes..." >> "$LOG"
    sleep 300
  fi

  if [ "$attempt" -eq 6 ]; then
    echo "❌ GitHub still unreachable after 30 minutes. Exiting." >> "$LOG"
    exit 1
  fi
done

# Step 1: Check for catalog sync error
ERROR_MSG=$(midclt call alert.list | grep -i 'Failed to sync TRUENAS catalog')
if [ -z "$ERROR_MSG" ]; then
  echo "✅ No catalog sync error detected. No action needed." >> "$LOG"
  exit 0
else
  echo "❗ Catalog sync error detected. Proceeding with fix..." >> "$LOG"
fi

# Step 2: Unset the app pool
echo "🔧 Unsetting Kubernetes app pool..." >> "$LOG"
midclt call kubernetes.update '{ "pool": null }' >> "$LOG" 2>&1
sleep 3

# Step 3: Reset the app pool
echo "🔧 Re-setting Kubernetes app pool to 'storage'..." >> "$LOG"
midclt call kubernetes.update '{ "pool": "storage" }' >> "$LOG" 2>&1
sleep 5

# Step 4: Trigger catalog refresh
echo "🔄 Triggering app catalog sync..." >> "$LOG"
midclt call catalog.sync >> "$LOG" 2>&1
sleep 5

# Step 5: Check for persistent error
POST_ERROR=$(midclt call alert.list | grep -i 'Failed to sync TRUENAS catalog')
if [ -z "$POST_ERROR" ]; then
  echo "✅ Catalog sync succeeded after reset." >> "$LOG"
else
  echo "❌ Catalog sync error still present after reset." >> "$LOG"
fi

# Step 6: Sanity check - ensure .ix-apps is writable
if mount | grep -q '/mnt/.ix-apps'; then
  if [ ! -w /mnt/.ix-apps ]; then
    echo "❌ .ix-apps is mounted but not writable!" >> "$LOG"
    echo "📋 ZFS pool status:" >> "$LOG"
    zpool status >> "$LOG"
  else
    echo "✅ .ix-apps is writable." >> "$LOG"
  fi
else
  echo "⚠️ .ix-apps is not mounted!" >> "$LOG"
fi

echo "🏁 Script complete." >> "$LOG"
1 Like

:white_check_mark: No Script Solution

After a lot of testing, I found the root cause of the recurring
[EFAULT] Failed to clone ... /mnt/.ix-apps/truenas_catalog errors. After correcting the root case, the script was no longer required.

In my case, the problem was ultimately a replication tasks copying the .ix-apps dataset to a backup pool, and causing the issue.

After replication, the backup pool’s copy of .ix-apps was being auto-mounted read-only under /mnt/.ix-apps, overlapping with the live application dataset.
That caused the app catalog clone to fail, which would only clear after a reboot.


:gear: Root Cause

When replicating the ix-apps dataset, TrueNAS also replicates its mountpoint and canmount properties. Once the backup pool receives those properties, SCALE tries to mount the replicated dataset automatically — creating read-only mounts like:

storage_backup/ix-apps/truenas_catalog on /mnt/.ix-apps/truenas_catalog (ro,noatime,...)

This directly conflicts with the live /mnt/.ix-apps location used by Apps and the catalog sync process.


:adhesive_bandage: Solution Steps

The simplest and most reliable solution is to exclude the conflicting ZFS properties from replication. This prevents the backup dataset from auto-mounting, while keeping the full dataset intact for restore.

Steps:

  1. Open your replication task (e.g. storage → storage_backup).
  2. Scroll down to Properties Exclude.
  3. Add the following as separate tags:
    • mountpoint
    • canmount
  4. Save the replication task.

This ensures that the replicated .ix-apps dataset on the backup pool stays unmounted and doesn’t interfere with the live one.


:arrows_counterclockwise: If You Already Replicated Before Adding the Exclusions

If you’ve already run replication before applying the exclusions, your backup .ix-apps datasets may still be mounted.
A reboot will temporarily unmount them, and the system will work until the next replication task runs.
However, the most reliable fix is to manually unmount and disable those mounts permanently.

Run these commands in Shell:

# 1. Unmount any backup .ix-apps datasets
for mp in $(mount | awk '/storage_backup\/ix-apps/ {print $3}' | sort -r); do
  echo "Unmounting $mp"
  sudo umount -f "$mp" || true
done

# 2. Prevent automatic mounting on the backup pool
sudo zfs list -H -r -o name -t filesystem storage_backup/ix-apps | while read ds; do
  sudo zfs set canmount=off "$ds"
  sudo zfs set mountpoint=none "$ds"
done

# 3. Refresh mounts
sudo zfs mount -a

After this, the fix is persistent — even after future replications or reboots.


:mag: Verification

You can confirm the fix by running the following command in Shell:

mount | grep '\.ix-apps'

After replication, only your primary pool’s .ix-apps datasets should appear mounted — the backups will remain inert.

If you want to check the ZFS properties directly, you can use:

zfs get mountpoint,canmount storage/ix-apps
zfs get mountpoint,canmount storage_backup/ix-apps

You should see canmount=off and mountpoint=none (or inherited) on the backup dataset.


:white_check_mark: Results

  • The app catalog now syncs normally and consistently.
  • No more [EFAULT] Failed to clone ... errors.
  • No need to reboot after replication or manually clear mounts.

Hopefully this helps others running into the same issue — especially if you’re replicating your Apps dataset between pools.

2 Likes

Interesting, thank you for taking your time to look into this!

Can you file an official Bug Report using the TrueNAS UI?
You will need to make a Jira account, but the issue with a replicated ix-apps triggering the fault likely affects others as well.

Sure thing. I’ll submit a bug report.

Please post a link to the JIRA ticket in this thread

[NAS-137899] Replicating .ix-apps Dataset Causes Auto-Mount Conflict and App Catalog Sync Failure - iXsystems TrueNAS Jira

1 Like