Circling back, the issue here appears to be that the debug kernel was enabled (System → Advanced → Kernel)
next step now 
root@truenas[~]# nvidia-smi
Fri Aug 1 18:37:02 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.142 Driver Version: 550.142 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Quadro P400 Off | 00000000:3B:00.0 Off | N/A |
| 28% 39C P0 N/A / N/A | 0MiB / 2048MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
root@truenas[~]#
on the other hand if I check the gpu box in my plex apps during deployment I get an error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 515, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 562, in __run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 627, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 294, in nf
rv = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 101, in wrapped
result = func(*args)
^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 229, in do_update
app = self.update_internal(job, app, data, trigger_compose=app['state'] != 'STOPPED')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 259, in update_internal
update_app_config(app_name, app['version'], new_values, custom_app=app['custom_app'])
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 60, in update_app_config
render_compose_templates(
File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 51, in render_compose_templates
raise CallError(f'Failed to render compose templates: {cp.stderr}')
middlewared.service_exception.CallError: [EFAULT] Failed to render compose templates: Traceback (most recent call last):
File "/usr/bin/apps_render_app", line 33, in <module>
sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 48, in main
render_templates_from_path(args.path, args.values)
File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path
rendered_data = render_templates(
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 31, in render_templates
rendered_templates[i.name] = env.get_template(i.name).render(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "/mnt/.ix-apps/app_configs/plex/versions/1.2.5/templates/docker-compose.yaml", line 3, in top-level template code
{% set c1 = tpl.add_container(values.consts.plex_container_name, values.plex.image_selector) %}
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/.ix-apps/app_configs/plex/versions/1.2.5/templates/library/base_v2_1_35/render.py", line 63, in add_container
container = Container(self, name, image)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/.ix-apps/app_configs/plex/versions/1.2.5/templates/library/base_v2_1_35/container.py", line 99, in __init__
self.deploy: Deploy = Deploy(self._render_instance)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/.ix-apps/app_configs/plex/versions/1.2.5/templates/library/base_v2_1_35/deploy.py", line 15, in __init__
self.resources: Resources = Resources(self._render_instance)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/.ix-apps/app_configs/plex/versions/1.2.5/templates/library/base_v2_1_35/resources.py", line 24, in __init__
self._auto_add_gpus_from_values()
File "/mnt/.ix-apps/app_configs/plex/versions/1.2.5/templates/library/base_v2_1_35/resources.py", line 55, in _auto_add_gpus_from_values
raise RenderError(f"Expected [uuid] to be set for GPU in slot [{pci}] in [nvidia_gpu_selection]")
base_v2_1_35.error.RenderError: Expected [uuid] to be set for GPU in slot [0000:3b:00.0] in [nvidia_gpu_selection]
