Help needed: Misconfigured image reference in custom app broke... many things

Apps and Virtualization
1

I tried to set up gitea as a custom app on Dragonfish-24.04.2, but copy-pasted a wrong docker image reference (gitea/gitea:@version@ instead of gitea/gitea) into the corresponding textbox in the web ui.
Since then, significant parts of TrueNAS Scale seem broken, not only can’t the apps service start anymore, I can’t even delete datasets anymore - everything fails with the following error:

[EFAULT] Invalid reference format: gitea/gitea:@version@:latest

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 47, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/catalogs_linux/apps.py", line 18, in latest
    await self.middleware.call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1428, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 191, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 53, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/catalogs_linux/apps.py", line 62, in available
    for app in self.middleware.call_sync('chart.release.query')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1589, in call_sync
    return self.run_coroutine(methodobj(*prepared_call.args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in run_coroutine
    return fut.result()
           ^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 47, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 241, in query
    list(set(
         ^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 242, in <genexpr>
    normalize_reference(c['image'])['complete_tag']
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/container_runtime_interface/utils.py", line 48, in normalize_reference
    raise CallError(f'Invalid reference format: {tagged_image}')
middlewared.service_exception.CallError: [EFAULT] Invalid reference format: gitea/gitea:@version@:latest

I have no idea how to fix that broken configuration, since I can’t access it via the web ui anymore - due to the error above.
Are there alternative ways, e.g. via CLI, to change the configuration for custom apps?
Any help would be greatly appreciated!

2

There may be a cli way, not sure, but that’s a bug if that is the case! Definitely report it via the bug reporting system and maybe they can help you find a workaround.

3

I reported a bug about this, but luckily I also found a workaround by now, after trying tons of things that didn’t help.
It’s an ugly hack that I definitely can’t recommend as a way to solve such problems, but I’ll report it here anyway in case anyone runs into the same issue and is as desperate to fix it as I.

Basically I edited the python code that throws the error and added some special treatment for my particular problem.

To get write access to the file in question:

mount -o remount,rw /usr

Edit it via

vi /usr/lib/python3/dist-packages/middlewared/plugins/container_runtime_interface/utils.py

Important: if you mix tabs and spaces in this file, middlewared crashes on the next reboot (learned that the hard way)! So make sure you only indent with spaces! (use :set list in vi so you’re able to see the difference)

Then, in the normalize_reference method I added the line

tagged_image = tagged_image.replace("gitea/gitea:@version@:latest","gitea/gitea:latest")

directly before the first read access to that variable, in order to repair the specific broken string that caused my issues.

Saved the file, rebooted (probably restarting the middlewared service would have been enough, but remember to remount /usr as read-only if you go that way), and the error was gone and my apps worked the way they’re supposed to. It looked like my gitea app was even starting successfully, but I didn’t look into it in more detail, I just deleted the app as soon as I had the chance.

I hope the TrueNAS team adds some additional safeguards for such situations (I’ll keep the bug ticket open of course), so that a single misconfigured app can’t take half of the system down with it, and I also hope for some more straightforward and less risky recovery mechanisms.

1 Like