I originally used the 24.04 system, and then I heard that the 24.10 version based on the Docker container was launched, so I upgraded it, but now I have a problem. My underlying environment is ESXI8.0 virtualized 24.10, and then I pass the graphics card and HBA card directly to the system. Now it can only recognize VM SVGA, and there is no output when executing Nvidia-smi in the command line. Can anyone help me?
Hey @Go_blin
Do you see the Tesla device in the lspci output? I’m not aware of anything saying they’ve dropped Pascal support. I don’t have that exact setup, but do have a P4 - so I can try to test 24.10 with it.
Just to be clear - you’re passing the entire P4 from ESXi via PCIe passthrough to the TrueNAS VM, and you’re not trying to introduce vGPU/GRID at any stage?
@HoneyBadger Yes, the P4 graphics driver cannot be seen in the lspci output, but the card can be seen, and nvidia-smi has no output.
I’ll have to give a baremetal EE install a shot in a bit.
Can you show me the output of lspci -v - is the GP104GL being picked up by the nvidia driver or is it failing to bind for some reason?
Can you capture the lower section of the output (or use an SSH session to the TrueNAS VM?) and ensure that it’s showing the nvidia family for driver/modules, and not something like vfio?
No, it hasn’t–that version is still in pre-release status, and is expected to be released in October.
Ah, then why can I experience it in advance, but the GPU driver is not loaded.
Starting in 24.10, we are no longer including a default Nvidia driver. This is to allow it to be updated separately from TrueNAS releases.
There will be a UI selectable option available to install/update nvidia drivers before 24.10.0, but for now you can set it using midclt call -job docker.update '{"nvidia": true}'.
4 Likes
Yes, thank you very much for your help. It has been identified, but some containers cannot be installed. It may be an adaptation problem. Please adapt it as soon as possible.
can you show the error message you received?
error
FAILED
[EFAULT] Failed to render compose templates: Traceback (most recent call last): File “/usr/bin/apps_render_app”, line 33, in sys.exit(load_entry_point(‘apps-validation==0.1’, ‘console_scripts’, ‘apps_render_app’)()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py”, line 47, in main render_templates_from_path(args.path, args.values) File “/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py”, line 19, in render_templates_from_path rendered_data = render_templates( ^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/catalog_templating/render.py”, line 36, in render_templates ).render({‘ix_lib’: template_libs, ‘values’: test_values}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/jinja2/environment.py”, line 1301, in render self.environment.handle_exception() File “/usr/lib/python3/dist-packages/jinja2/environment.py”, line 936, in handle_exception
What were you attempting to install at the time and how? Via a catalog app in the UI, via CLI, etc?
FAILED
[EFAULT] Failed to render compose templates: Traceback (most recent call last): File "/usr/bin/apps_render_app", line 33, in sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 47, in main render_templates_from_path(args.path, args.values) File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path rendered_data = render_templates( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 36, in render_templates ).render({'ix_lib': template_libs, 'values': test_values}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render self.environment.handle_exception() File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "/mnt/.ix-apps/app_configs/qbittorrent/versions/1.0.10/templates/docker-compose.yaml", line 64, in top-level template code }, user=values.qbittorrent.additional_envs, values=values) | tojson }} ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 485, in getattr return getattr(obj, attribute) ^^^^^^^^^^^^^^^^^^^^^^^ jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'qbittorrent'
remove_circle_outline
More info…
错误: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 513, in __run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1363, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1360, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 268, in nf
rv = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
res = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 195, in do_create
return self.create_internal(job, app_name, version, data['values'], complete_app_details)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 244, in create_internal
raise e from None
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 224, in create_internal
update_app_config(app_name, version, new_values)
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 57, in update_app_config
render_compose_templates(
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 48, in render_compose_templates
raise CallError(f'Failed to render compose templates: {cp.stderr}')
middlewared.service_exception.CallError: [EFAULT] Failed to render compose templates: Traceback (most recent call last):
File "/usr/bin/apps_render_app", line 33, in <module>
sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 47, in main
render_templates_from_path(args.path, args.values)
File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path
rendered_data = render_templates(
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 36, in render_templates
).render({'ix_lib': template_libs, 'values': test_values})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "/mnt/.ix-apps/app_configs/qbittorrent/versions/1.0.10/templates/docker-compose.yaml", line 64, in top-level template code
}, user=values.qbittorrent.additional_envs, values=values) | tojson }}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 485, in getattr
return getattr(obj, attribute)
^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'qbittorren
[EFAULT] Failed to render compose templates: Traceback (most recent call last): File "/usr/bin/apps_render_app", line 33, in sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 47, in main render_templates_from_path(args.path, args.values) File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path rendered_data = render_templates( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 36, in render_templates ).render({'ix_lib': template_libs, 'values': test_values}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render self.environment.handle_exception() File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "/mnt/.ix-apps/app_configs/immich/versions/1.0.5/templates/docker-compose.yaml", line 36, in top-level template code {% if values.immich.enable_ml %} ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 485, in getattr return getattr(obj, attribute) ^^^^^^^^^^^^^^^^^^^^^^^ jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'immich'
remove_circle_outline
More info…
错误: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 513, in __run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1363, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1360, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 268, in nf
rv = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
res = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 195, in do_create
return self.create_internal(job, app_name, version, data['values'], complete_app_details)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 244, in create_internal
raise e from None
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 224, in create_internal
update_app_config(app_name, version, new_values)
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 57, in update_app_config
render_compose_templates(
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 48, in render_compose_templates
raise CallError(f'Failed to render compose templates: {cp.stderr}')
middlewared.service_exception.CallError: [EFAULT] Failed to render compose templates: Traceback (most recent call last):
File "/usr/bin/apps_render_app", line 33, in <module>
sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 47, in main
render_templates_from_path(args.path, args.values)
File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path
rendered_data = render_templates(
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 36, in render_templates
).render({'ix_lib': template_libs, 'values': test_values})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "/mnt/.ix-apps/app_configs/immich/versions/1.0.5/templates/docker-compose.yaml", line 36, in top-level template code
{% if values.immich.enable_ml %}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 485, in getattr
return getattr(obj, attribute)
^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'immich'
I’d suggest submitting a bug ticket at this point with a full description of which apps you were attempting to install and how. There is a Report a Bug link if you scroll to the top of this page. After you submit an automatic comment will link you to the private upload service to attach a debug file from the system.
@DjP-iX Hello sir. There are same problem aslo happened to me.
I’m using nvidia P40 with ESXI8.0 & TrueNAS 24.10.
I installed the nvidia vGPU/GRID driver, and the nvidia-smi test result aslo is postive.
At the application setting page, I can’t find the Install nvidia driver check box.
At the plex configuration page, I can see the GPU.
But I try to enter the plex container shell, I can’t find the cmd nvidia-smi…
Could you give me some advance please?
Hi @usamimizugi. We’re investigating a known issue with Plex GPU allocation and plan to have a fix for it included in 24.10.0.
2 Likes
I’m running latest EE-RC2, with a fresh install of Plex and I have a Tesla P4. Nvidia-SMI tells me from inside the plex container I have a Tesla P4 so I recon this then means all is good on the driver part.
Nvidia-Smi shows me, see below, I’m decoding with hw, but as soon as I try hw encode - it just breaks and never plays:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P4 Off | 00000000:82:00.0 Off | 0 |
| N/A 74C P0 26W / 75W | 167MiB / 7680MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 653613 C ...lib/plexmediaserver/Plex Transcoder 164MiB |
+-----------------------------------------------------------------------------------------+
It’s when I checks “Use hardware-accelerated video encoding” in the Plex Transcoder settings that things break. If that’s unchecked I only hw decode.
Is this also related to what will be fixed in 24.10.0?
EDIT: Btw this is even if I got an error during installation of the GPU drivers under apps->config->settings:
Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 488, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 535, in __run_body rv = await self.middleware.run_in_thread(self.method, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1363, in run_in_thread return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1360, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/nvidia.py", line 65, in install self._install_driver(job, td, path)
File "/usr/lib/python3/dist-packages/middlewared/plugins/nvidia.py", line 133, in _install_driver subprocess.run([path, "--tmpdir", td, "-s"], capture_output=True, check=True, text=True)
File "/usr/lib/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/root/tmpjbbxiz7i/NVIDIA-Linux-x86_64-550.120-no-compat32.run', '--tmpdir', '/root/tmpjbbxiz7i', '-s']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 488, in run await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 533, in __run_body rv = await self.method(*args) ^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf res = await f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py", line 106, in do_update await ( File "/usr/lib/python3/dist-packages/middlewared/job.py", line 436, in wait raise self.exc_info[1]
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 492, in run raise handled middlewared.service_exception.CallError: [EFAULT] Command /root/tmpjbbxiz7i/NVIDIA-Linux-x86_64-550.120-no-compat32.run --tmpdir /root/tmpjbbxiz7i -s failed (code 1): Verifying archive integrity... OK Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 550.120.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
ERROR: Unable to load the kernel module 'nvidia.ko'. This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or if another driver, such as nouveau, is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA device(s), or no NVIDIA device installed in this system is supported by this NVIDIA Linux graphics driver release. Please see the log entries 'Kernel module load error' and 'Kernel messages' at the end of the file '/var/log/nvidia-installer.log' for more information. ERROR: Installation has failed. Please see the file '/var/log/nvidia-installer.log' for details. You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.
In the Plex App, I get this ambiguously namned GPU to select from still
I have a P400 and having the same issue trying to run Ollama
[EFAULT] Failed to render compose templates: Traceback (most recent call last): File “/usr/bin/apps_render_app”, line 33, in sys.exit(load_entry_point(‘apps-validation==0.1’, ‘console_scripts’, ‘apps_render_app’)()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py”, line 47, in main render_templates_from_path(args.path, args.values) File “/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py”, line 19, in render_templates_from_path rendered_data = render_templates( ^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/catalog_templating/render.py”, line 36, in render_templates ).render({‘ix_lib’: template_libs, ‘values’: test_values}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/jinja2/environment.py”, line 1301, in render self.environment.handle_exception() File “/usr/lib/python3/dist-packages/jinja2/environment.py”, line 936, in handle_exception raise rewrite_traceback_stack(source=source) File “/mnt/.ix-apps/app_configs/ollama/versions/1.0.0/templates/docker-compose.yaml”, line 3, in top-level template code {% set c1 = tpl.add_container(values.consts.ollama_container_name, “image”) %} ^^^^^^^^^^^^^^^^^^^^^^^^^ File “/mnt/.ix-apps/app_configs/ollama/versions/1.0.0/templates/library/base_v2_0_0/render.py”, line 50, in add_container container = Container(self, name, image) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/mnt/.ix-apps/app_configs/ollama/versions/1.0.0/templates/library/base_v2_0_0/container.py”, line 58, in init self.deploy: Deploy = Deploy(self._render_instance) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/mnt/.ix-apps/app_configs/ollama/versions/1.0.0/templates/library/base_v2_0_0/deploy.py”, line 15, in init self.resources: Resources = Resources(self._render_instance) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/mnt/.ix-apps/app_configs/ollama/versions/1.0.0/templates/library/base_v2_0_0/resources.py”, line 24, in init self._auto_add_gpus_from_values() File “/mnt/.ix-apps/app_configs/ollama/versions/1.0.0/templates/library/base_v2_0_0/resources.py”, line 55, in _auto_add_gpus_from_values raise RenderError(f"Expected [uuid] to be set for GPU in slot [{pci}] in [nvidia_gpu_selection]") base_v2_0_0.error.RenderError: Expected [uuid] to be set for GPU in slot [0000:00:1c.0] in [nvidia_gpu_selection]