Script to Fix App Catalog Sync Errors

I setup a Terramaster F8 Plus with TrueNAS Scale 25.04.1, and was experiencing app catalog sync errors that I couldn’t fix via the web interface. Others solved the issue by setting the bios time or manually unsetting/setting the app pool, but those didn’t work well for me. I think the main issue is how TrueNAS handles the network bridge I have setup.

To resolve the app catalog sync issues permanently, I created a script that manually syncs the app catalog via a cron job, checks that certain conditions are in place, and handles common errors automatically. After a lot of work, and some help from ChatGPT, I came up with the following to fix the app sync errors.

Problem:
If you’re encountering this recurring error on TrueNAS SCALE:

[EFAULT] Failed to clone ‘GitHub - truenas/apps’ repository at ‘/mnt/.ix-apps/truenas_catalog’ destination…

…it often prevents you from syncing the app catalog, especially during scheduled tasks. In some cases, the issue persists until the apps pool is unset and reset manually in the UI.

Root Cause (Suspected):

  • Having a network bridge setup appears to trigger the issue for me.
  • Catalog sync errors seem to be caused by the /mnt/.ix-apps/truenas_catalog dataset being in a weird mount state or the underlying middleware process getting jammed.
  • Manually unsetting and resetting the Apps pool in the UI always resolves the issue — suggesting an underlying bug in how the system handles app catalog mounts during sync.

Solution:
The solution is a script that checks for the catalog error, and if it exists:

  1. Unsets the app pool.
  2. Waits a few seconds.
  3. Resets the same app pool.
  4. Syncs the catalog.
  5. Logs actions and timestamps to /var/log/fix_ix_apps.log.

Directions:

To create the script, you will need to use shell. Shell can be accessed either from the TrueNAS web UI (Shell tab) or by connecting through SSH (e.g., with Termux, JuiceSSH, or PuTTY).

  1. Open Nano to Create the Script
sudo nano /root/fix_ix_apps.sh
  1. Paste This Script

Once nano is open, add the script text below.

#!/bin/bash

# ========================
# TrueNAS App Catalog Fix Script
# ========================
# ✅ Checks for TRUENAS catalog sync errors
# ✅ If present, unsets and resets the Kubernetes app pool
# ✅ Triggers catalog refresh
# ✅ Verifies success and logs outcome
# ✅ Includes .ix-apps sanity check
# ✅ Retries GitHub check up to 30 minutes before proceeding
# ========================

# Set log path
LOG="/var/log/fix_ix_apps.log"

# Timestamp
echo "=== $(date) ===" >> "$LOG"
echo "🔧 Starting TrueNAS app catalog fix script..." >> "$LOG"

# Step 0: Check if GitHub is reachable (retry for up to 30 minutes)
for attempt in {1..6}; do
  if curl -Is https://github.com --max-time 10 >/dev/null 2>&1; then
    echo "🌐 GitHub is reachable." >> "$LOG"
    break
  else
    echo "⏳ GitHub unreachable. Attempt $attempt/6. Retrying in 5 minutes..." >> "$LOG"
    sleep 300
  fi

  if [ "$attempt" -eq 6 ]; then
    echo "❌ GitHub still unreachable after 30 minutes. Exiting." >> "$LOG"
    exit 1
  fi
done

# Step 1: Check for catalog sync error
ERROR_MSG=$(midclt call alert.list | grep -i 'Failed to sync TRUENAS catalog')
if [ -z "$ERROR_MSG" ]; then
  echo "✅ No catalog sync error detected. No action needed." >> "$LOG"
  exit 0
else
  echo "❗ Catalog sync error detected. Proceeding with fix..." >> "$LOG"
fi

# Step 2: Unset the app pool
echo "🔧 Unsetting Kubernetes app pool..." >> "$LOG"
midclt call kubernetes.update '{ "pool": null }' >> "$LOG" 2>&1
sleep 3

# Step 3: Reset the app pool
echo "🔧 Re-setting Kubernetes app pool to 'storage'..." >> "$LOG"
midclt call kubernetes.update '{ "pool": "storage" }' >> "$LOG" 2>&1
sleep 5

# Step 4: Trigger catalog refresh
echo "🔄 Triggering app catalog sync..." >> "$LOG"
midclt call catalog.sync >> "$LOG" 2>&1
sleep 5

# Step 5: Check for persistent error
POST_ERROR=$(midclt call alert.list | grep -i 'Failed to sync TRUENAS catalog')
if [ -z "$POST_ERROR" ]; then
  echo "✅ Catalog sync succeeded after reset." >> "$LOG"
else
  echo "❌ Catalog sync error still present after reset." >> "$LOG"
fi

# Step 6: Sanity check - ensure .ix-apps is writable
if mount | grep -q '/mnt/.ix-apps'; then
  if [ ! -w /mnt/.ix-apps ]; then
    echo "❌ .ix-apps is mounted but not writable!" >> "$LOG"
    echo "📋 ZFS pool status:" >> "$LOG"
    zpool status >> "$LOG"
  else
    echo "✅ .ix-apps is writable." >> "$LOG"
  fi
else
  echo "⚠️ .ix-apps is not mounted!" >> "$LOG"
fi

echo "🏁 Script complete." >> "$LOG"
  1. Save and Exit in Nano
  • Press Ctrl + O to write the file
  • Press Enter to confirm the filename
  • Press Ctrl + X to exit nano
  1. (Optional) Schedule the Script to Run Automatically

To make this permanent, you will need to setup a cron job. Otherwise, you will need to run the script manually. The command below will make the script run automatically.

chmod +x /root/fix_ix_apps.sh
  1. Check the Script’s Log

To verify the script is running as intended, check the log file. ChatGPT is great at interpreting the log. The command below will show you the log details.

tail -n 50 /var/log/fix_ix_apps.log

Final Thoughts:

This script is working well for me. If you run into issues with the script, feel free to post your issue or modified script here.

1 Like

Excellent, been putting off making something like this for myself as I have weird issues with my bridge only on boot specifically.

Thoughts on doing a check on if github is reachable prior to the un/reset? Otherwise not much point imo

That would be a good addition.

If github were unreachable, it could wait a while and checking again.

1 Like

I updated the script so it now checks if github is reachable.

#!/bin/bash

# ========================
# TrueNAS App Catalog Fix Script
# ========================
# ✅ Checks for TRUENAS catalog sync errors
# ✅ If present, unsets and resets the Kubernetes app pool
# ✅ Triggers catalog refresh
# ✅ Verifies success and logs outcome
# ✅ Includes .ix-apps sanity check
# ✅ Retries GitHub check up to 30 minutes before proceeding
# ========================

# Set log path
LOG="/var/log/fix_ix_apps.log"

# Timestamp
echo "=== $(date) ===" >> "$LOG"
echo "🔧 Starting TrueNAS app catalog fix script..." >> "$LOG"

# Step 0: Check if GitHub is reachable (retry for up to 30 minutes)
for attempt in {1..6}; do
  if curl -Is https://github.com --max-time 10 >/dev/null 2>&1; then
    echo "🌐 GitHub is reachable." >> "$LOG"
    break
  else
    echo "⏳ GitHub unreachable. Attempt $attempt/6. Retrying in 5 minutes..." >> "$LOG"
    sleep 300
  fi

  if [ "$attempt" -eq 6 ]; then
    echo "❌ GitHub still unreachable after 30 minutes. Exiting." >> "$LOG"
    exit 1
  fi
done

# Step 1: Check for catalog sync error
ERROR_MSG=$(midclt call alert.list | grep -i 'Failed to sync TRUENAS catalog')
if [ -z "$ERROR_MSG" ]; then
  echo "✅ No catalog sync error detected. No action needed." >> "$LOG"
  exit 0
else
  echo "❗ Catalog sync error detected. Proceeding with fix..." >> "$LOG"
fi

# Step 2: Unset the app pool
echo "🔧 Unsetting Kubernetes app pool..." >> "$LOG"
midclt call kubernetes.update '{ "pool": null }' >> "$LOG" 2>&1
sleep 3

# Step 3: Reset the app pool
echo "🔧 Re-setting Kubernetes app pool to 'storage'..." >> "$LOG"
midclt call kubernetes.update '{ "pool": "storage" }' >> "$LOG" 2>&1
sleep 5

# Step 4: Trigger catalog refresh
echo "🔄 Triggering app catalog sync..." >> "$LOG"
midclt call catalog.sync >> "$LOG" 2>&1
sleep 5

# Step 5: Check for persistent error
POST_ERROR=$(midclt call alert.list | grep -i 'Failed to sync TRUENAS catalog')
if [ -z "$POST_ERROR" ]; then
  echo "✅ Catalog sync succeeded after reset." >> "$LOG"
else
  echo "❌ Catalog sync error still present after reset." >> "$LOG"
fi

# Step 6: Sanity check - ensure .ix-apps is writable
if mount | grep -q '/mnt/.ix-apps'; then
  if [ ! -w /mnt/.ix-apps ]; then
    echo "❌ .ix-apps is mounted but not writable!" >> "$LOG"
    echo "📋 ZFS pool status:" >> "$LOG"
    zpool status >> "$LOG"
  else
    echo "✅ .ix-apps is writable." >> "$LOG"
  fi
else
  echo "⚠️ .ix-apps is not mounted!" >> "$LOG"
fi

echo "🏁 Script complete." >> "$LOG"
1 Like