ElectricEel-24.10.1 # Failed to sync TRUENAS catalog

Hello,
today I received the following critical alarm:

Failed to sync TRUENAS catalog: [EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: [EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: Cloning into '/mnt/.ix-apps/truenas_catalog'......

A manually tried to refresh the apps catalog, failed also with the following message:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/catalog/git_utils.py", line 34, in pull_clone_repository
    clone_repository(repository_uri, destination, branch, depth)
  File "/usr/lib/python3/dist-packages/middlewared/utils/git.py", line 25, in clone_repository
    raise CallError(
middlewared.service_exception.CallError: [EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: Cloning into '/mnt/.ix-apps/truenas_catalog'......

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 509, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 554, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/catalog/sync.py", line 27, in sync
    await self.middleware.call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1471, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/catalog/sync.py", line 55, in update_git_repository
    return pull_clone_repository(repository, location, branch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/catalog/git_utils.py", line 36, in pull_clone_repository
    raise CallError(f'Failed to clone {repository_uri!r} repository at {destination!r} destination: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: [EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: Cloning into '/mnt/.ix-apps/truenas_catalog'......

Thy system is now running since 7 days. In this time I updated several docker apps without a problem.
Nothing has changed during that time and the NAS has still internet connection. Also the system time is correct and NTP is running:

root@TrueNAS[~]# timedatectl
               Local time: Tue 2025-01-21 20:10:49 CET
           Universal time: Tue 2025-01-21 19:10:49 UTC
                 RTC time: Tue 2025-01-21 19:10:49
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
root@TrueNAS[~]# ping www.google.com
PING www.google.com (172.217.16.196) 56(84) bytes of data.
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=1 ttl=118 time=27.6 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=2 ttl=118 time=27.8 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=3 ttl=118 time=27.5 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=4 ttl=118 time=27.8 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=5 ttl=118 time=27.7 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=6 ttl=118 time=28.8 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=7 ttl=118 time=28.6 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=8 ttl=118 time=28.9 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=9 ttl=118 time=28.6 ms
64 bytes from fra16s65-in-f4.1e100.net (172.217.16.196): icmp_seq=10 ttl=118 time=28.5 ms
--- www.google.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 10274ms
rtt min/avg/max/mdev = 27.545/28.172/28.868/0.498 ms
root@TrueNAS[~]# 

Any suggestions what causes this behavior aand how to fix it?
Thanks for you help.

Chris

Can you ping GitHub successfully from the shell?

1 Like

Unfortunately I did a restart to the system and now the alert is gone and everything seems to work as expected. I will observe the system. But I did also a ping to github.com, passed.

root@TrueNAS[~]# ping github.com
PING github.com (140.82.121.4) 56(84) bytes of data.
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=1 ttl=54 time=42.1 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=2 ttl=54 time=29.1 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=3 ttl=54 time=27.5 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=4 ttl=54 time=28.8 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=5 ttl=54 time=27.7 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=6 ttl=54 time=30.7 ms
github.com ping statistics —
6 packets transmitted, 6 received, 0% packet loss, time 5007ms
rtt min/avg/max/mdev = 27.540/30.993/42.112/5.079 ms
root@TrueNAS[~]#

1 Like

Glad to hear the issue has gone.

we will see for how long.
But a reboot is not really a proper solution