TrueNAS Apps catalog fails to sync - Git HTTP/2 returns 401, HTTP/1.1 works

Apps and Virtualization
, ,
1

Hi everyone,

I’m posting this mainly for future reference in case anyone else runs into the same issue. I’m not a programmer or sysadmin, and this investigation was done with the help of ChatGPT and Claude, so I’m sharing what I found rather than claiming to know the root cause. This post was also written with the help of an LLM.

Today, my TrueNAS Apps catalog suddenly stopped syncing. The system had been running the same TrueNAS build (25.10.6) since the beginning of August, and I hadn’t made any changes immediately beforehand.

The error was:

Failed to sync catalog. Please try clicking "Refresh Catalog" manually.

[EFAULT] Failed to clone 'https://github.com/truenas/apps'
repository at '/mnt/.ix-apps/truenas_catalog' destination

Initially, I suspected a network/DNS problem or something wrong with the Apps dataset.

The ZFS pool was healthy and all the Apps datasets were still present. GitHub was also reachable normally:

curl -I https://github.com/truenas/apps
→ HTTP 200

The Git smart HTTP endpoint also returned HTTP 200. However, Git itself was behaving strangely. A normal clone resulted in an authentication prompt / HTTP 401, despite the repository being public.

I checked the Git configuration, credentials, proxy settings, etc., and couldn’t find anything obviously wrong.

The interesting part

Forcing Git to use HTTP/1.1 made the problem disappear:

git -c http.version=HTTP/1.1 clone https://github.com/truenas/apps.git /tmp/test-clone

The entire ~267 MB repository cloned successfully.

I then set Git to use HTTP/1.1:

git config --system http.version HTTP/1.1

After that, I went back to Apps → Discover Apps → Refresh Catalog, and the TrueNAS catalog updated successfully.

Interestingly, I’ve now seen the same catalog problem on 25.10.5, 25.10.6 and 25.10.7. After upgrading to 25.10.7, the catalog failed again, and the same HTTP/1.1 workaround fixed it.

There was some additional Apps/Docker trouble while troubleshooting 25.10.6, but I don’t consider that evidence of a TrueNAS regression since I was also changing system configuration at the time. The original catalog problem was separate and reproducible.

What I don’t know

I have no idea whether this is a TrueNAS issue, Git/libcurl, GitHub, or something in the network path between my server and GitHub.

The behaviour I observed was essentially:

Git → GitHub via HTTP/2  →  HTTP 401 / fails
Git → GitHub via HTTP/1.1 → works
TrueNAS catalog           → works after forcing HTTP/1.1

I couldn’t find an obvious GitHub incident around September 2 that matched this.

If anyone has seen something similar, particularly GitHub returning a 401 over HTTP/2 while the same Git operation works over HTTP/1.1, I’d be interested to hear what might be causing it.

Hopefully this is useful to someone else before they start messing with their Apps datasets. :slight_smile:

12 Likes
TrueNAS 25.10.7 is Now Available
Script to Fix App Catalog Sync Errors
2

Ok… now, I know it’s not purely my fault.

I didn’t come so far with Gemini, but at least it tried to convince me that it would be a briliant idea to re-init the ix-apps dataset by destroying it.

So lessons leared - AI is like statistics… do not ever trust them…

regards

2 Likes
3

Hi OvrilPT,

Thanks for this info. My truenas system also not syncing the catalog, gave the same error.

But your solution works like a charme! I also set git to use http/1.1 as you do and boom, my catalog is updated successfully.

Now I am awaiting the truenas team to release a bug fix update for that. :slight_smile:

Thanks again, it helps in the meantime.

1 Like
4

This is going to sound stupid - but when I have catalogue failing to sync, but still have WAN access from the NAS, I can generally ‘fix’ it by making any arbitrary change in the networking tab & then reverting it.

Edit: I also have no explanation why this does the needful as bouncing the ports doesn’t. Maybe systemctl restart NetworkManager would do it? Will try next time.

1 Like
5

It’s happening to me too in 25.10.6 - Goldeye

6

Same problem 25.10.6 & 25.10.7

My solution:

git config --system http.version HTTP/1.1
cat /etc/gitconfig
1 Like
7

Today I updated from 25.10.4 → 25.10.7 and ran into this problem.

After applying your fix

sync of TRUENAS catalog is working again.

2 Likes
8

same problem with 25.10.7….will there be an official fix?

9

This is an issue with GitHub, go figure. It seems to be an issue with an older version of libcurl which is what the git binary pulls in as a dependency. I’m seeing this fail on versions of ubuntu as well. This isn’t a regression with 25.10.7 but seems to be something external to us. I’ve got no immediate solution.

5 Likes
TrueNAS 25.10.7 is Now Available
10

Same issue here. It started happening few hours ago. I landed on the same solution.

11

same to me, version 25.10.3. to 7 seem to have this problem. Thank you for this smart research!! Ki won´help this time

1 Like
12

It’s happenning because github recently stopped liking older git version on truenas (2.39.5).

Older client, HTTP/2 and git protocol version 2 (both default) trigger authentication on anonymous catalog refresh git clone, which fails as truenas doesn’t provide any credentials.

Swithing to HTTP/1.1 or older git protocol version both works around it, but proper fix is to update git on truenas.

13

Can you update git version on truenas? That fixes the issue (that very well might be old curl dependency behind).

14

Thanks for the information. I’ve finally been able to fix the problem with my Truenas 25.10.7 and my test NAS 25.10.6. After hours of struggling with the issue, I can finally sleep peacefully, knowing my machine is working normally. Many thanks. :sweat_smile:

1 Like
15

You truly are a bless men, same issue tried to update, din’t worked, talk to claude, gemini… nothing they were telling me it was probably a dns and told me to factory reset my router. Then I found you, thank you so much :heart_hands:t3:

1 Like
16

I believe the problem is actually simpler than this:
looks like GitHub started complaining when you try cloning without any auth.

Maybe your version trick works because they are rolling out this gradually, and this way you are slightly “faking” to be some edge/IoT device, and GitHub is deliberately still letting these devices clone stuff with no auth (?), to avoid breaking unmanned/unmaintained deployments.

Broke a couple of other unrelated things in my setup.
It was easy to connect the dots when I got the notification from TrueNAS.

To test this, just try to:
cd /tmp/
git clone ``https://github.com/truenas/apps

it will ask your GitHub user/pass.

If you instead:
git clone ``https://GH_USER:GH_PAT@github.com/truenas/apps

it will immediately work.

1 Like
17

ok, so we can all agree now, that Github is slowly changing on their server side to trigger Authentication issues.

Since TrueNAS is a managed OS, i do not wanna to modify around by changing git protocols. Can we atleast get a full confirmation from the TrueNAS team that this is an issue which will be fixed in a future patch? Just trying to bring some clarity here.

Thanks

18

After upgrading to 25.10.7 had the same problem. Switched to HTTP/1.1 as @Tom_G suggested and now it works. Thanks.

19
20

Reached the same result last night :)) really wish i had found this post first
Cheers!

1 Like