I’ve noticed a puzzling behavior in TrueNAS Scale 25.04.1: the REST API intermittently returns 401 “Unauthorized” errors when called rapidly or concurrently. This is affecting several clients and dashboard widgets including the Homepage TrueNAS widget.
Issue Description
In Scale 25.04.1, endpoints like /api/services/proxy sometimes return 401 Invalid API key even with a valid API key.
Triggered by parallel or fast consecutive requests – e.g. refreshing the Homepage widget twice quickly can yield one successful call and one 401 error .
This “random 401” happens even when using freshly created API keys, and also appears in custom clients calling directly via Python or curl .
Reproduction Steps
Create a new API key via TrueNAS UI/service account.
Send two identical API calls in quick succession (e.g., fetch /api/services/proxy):
First call returns 200 OK.
Second call returns 401.
Slow, spaced-out calls always succeed.
Confirmed across UI widgets, curl, and Python clients .
Impact
Front-end integration widgets like Homepage suffer flickering, JSON parse errors, and sometimes fail to load because of these transient 401 issues.
This appears to be undocumented, and with REST API marked as deprecated and, with no specific date in which they will stop being available, it’s unclear whether this behavior is a bug or an intentional step to force migration.
Known Workarounds
Throttle API calls (e.g., use 2–3-second delay between calls). Works, but clumsy.
Implement a buffering proxy or service to serialize API requests before they hit the TrueNAS REST API.
Switch to JSON-RPC / WebSocket APIs (SOAP-like). Some users built a Flask intermediary using auth.login_with_api_key, system.info, pool.dataset.details, etc.
Discussion Points
Has anyone else observed these intermittent 401 errors when using REST API in 25.04.1?
Are there existing TrueNAS-side fixes or guidance planned (besides “move to JSON-RPC”)?
Should widget/distributed client code implement retry/queue logic?
Would a community-maintained thin adapter layer (REST → JSON‑RPC/WebSocket) help ease transition?
Are there plans to restore deterministic REST authentication behavior or deprecate properly with documentation? (At least documentation about the authentication changes/impact would have been great)
Homepage maintainer here. Thanks for raising this question Wilferik, I certainly think this is most accurately described as a bug. Throttling of 2-3 seconds is not really acceptable for any api and even in the most charitable description of the behavior, there’s no way that a 401 “Invalid API key” is correct, as that is false.
Would love confirmation if this is going to be addressed or even the root cause.
Somewhat orthogonal to this issue, but it really seems like high time to either convert the widget to use the Websocket API, or develop a different widget that does–this API has been available since the first release of SCALE, and CORE is for all practical purposes long since dead.[1] And as you know, the REST API has been formally designated as deprecated with 25.04’s release.
That doesn’t excuse the behavior you’re seeing (which I’d agree is buggy), but if you already have a nasType field for the widget, it’s looking like time for a rewrite.
It hasn’t had any active development in five years other than a rebase to FreeBSD 13. ↩︎
I had feared this would be the answer. Honestly I’m not clear why a websocket api is replacing a rest http api but I’m sure they don’t really care about my thoughts on that. I’m also not clear on the timeline for actual removal.
Anyway, rewriting to use ws is not likely to be trivial and as you said, the http api is still there for now so it should work. We’re not asking for new development on it, we’re asking to… not break it
I tracked down the issue to a bug in the PAM module handling API key authentication. Should be fixed for 25.04.2. In theory could have impacted WS users as well, but would basically require spamming WS sessions. The REST API is layered on WS in such a way that each REST request temporarily creates an authenticated middleware session, which means it is much more likely to hit an issue.
I am on TrueNAS Scale Fangtooth 25.04.1 and this is also affecting the login page. It is intermittently and painful when logged into the console and suddenly connection is dropped. Didn’t have this issue before the update.
No. This bug wouldn’t affect the login page. It was very specific to API key handling. The bug ticket in general was a nice example of a helpful ticket from the community. It was clear and contained a simple repro case. This helps us to deliver fixes in a timely manner.
