Truenas Scale 24.10 cannot recognize Tesla P4 graphics card driver

I originally used the 24.04 system, and then I heard that the 24.10 version based on the Docker container was launched, so I upgraded it, but now I have a problem. My underlying environment is ESXI8.0 virtualized 24.10, and then I pass the graphics card and HBA card directly to the system. Now it can only recognize VM SVGA, and there is no output when executing Nvidia-smi in the command line. Can anyone help me?

Hey @Go_blin

Do you see the Tesla device in the lspci output? I’m not aware of anything saying they’ve dropped Pascal support. I don’t have that exact setup, but do have a P4 - so I can try to test 24.10 with it.

Just to be clear - you’re passing the entire P4 from ESXi via PCIe passthrough to the TrueNAS VM, and you’re not trying to introduce vGPU/GRID at any stage?

@HoneyBadger Yes, the P4 graphics driver cannot be seen in the lspci output, but the card can be seen, and nvidia-smi has no output.

I’ll have to give a baremetal EE install a shot in a bit.

Can you show me the output of lspci -v - is the GP104GL being picked up by the nvidia driver or is it failing to bind for some reason?

@HoneyBadger

Can you capture the lower section of the output (or use an SSH session to the TrueNAS VM?) and ensure that it’s showing the nvidia family for driver/modules, and not something like vfio?

No, it hasn’t–that version is still in pre-release status, and is expected to be released in October.

Ah, then why can I experience it in advance, but the GPU driver is not loaded.

Starting in 24.10, we are no longer including a default Nvidia driver. This is to allow it to be updated separately from TrueNAS releases.

There will be a UI selectable option available to install/update nvidia drivers before 24.10.0, but for now you can set it using midclt call -job docker.update '{"nvidia": true}'.

4 Likes

Yes, thank you very much for your help. It has been identified, but some containers cannot be installed. It may be an adaptation problem. Please adapt it as soon as possible.

@DjP-iX

can you show the error message you received?

@DjP-iX

error
FAILED

[EFAULT] Failed to render compose templates: Traceback (most recent call last): File “/usr/bin/apps_render_app”, line 33, in sys.exit(load_entry_point(‘apps-validation==0.1’, ‘console_scripts’, ‘apps_render_app’)()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py”, line 47, in main render_templates_from_path(args.path, args.values) File “/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py”, line 19, in render_templates_from_path rendered_data = render_templates( ^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/catalog_templating/render.py”, line 36, in render_templates ).render({‘ix_lib’: template_libs, ‘values’: test_values}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/jinja2/environment.py”, line 1301, in render self.environment.handle_exception() File “/usr/lib/python3/dist-packages/jinja2/environment.py”, line 936, in handle_exception

What were you attempting to install at the time and how? Via a catalog app in the UI, via CLI, etc?

@DjP-iX

FAILED
[EFAULT] Failed to render compose templates: Traceback (most recent call last): File "/usr/bin/apps_render_app", line 33, in sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 47, in main render_templates_from_path(args.path, args.values) File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path rendered_data = render_templates( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 36, in render_templates ).render({'ix_lib': template_libs, 'values': test_values}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render self.environment.handle_exception() File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "/mnt/.ix-apps/app_configs/qbittorrent/versions/1.0.10/templates/docker-compose.yaml", line 64, in top-level template code }, user=values.qbittorrent.additional_envs, values=values) | tojson }} ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 485, in getattr return getattr(obj, attribute) ^^^^^^^^^^^^^^^^^^^^^^^ jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'qbittorrent'
remove_circle_outline

More info…

 错误: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 513, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1363, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1360, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 268, in nf
    rv = func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 195, in do_create
    return self.create_internal(job, app_name, version, data['values'], complete_app_details)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 244, in create_internal
    raise e from None
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 224, in create_internal
    update_app_config(app_name, version, new_values)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 57, in update_app_config
    render_compose_templates(
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 48, in render_compose_templates
    raise CallError(f'Failed to render compose templates: {cp.stderr}')
middlewared.service_exception.CallError: [EFAULT] Failed to render compose templates: Traceback (most recent call last):
  File "/usr/bin/apps_render_app", line 33, in <module>
    sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 47, in main
    render_templates_from_path(args.path, args.values)
  File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path
    rendered_data = render_templates(
                    ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 36, in render_templates
    ).render({'ix_lib': template_libs, 'values': test_values})
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/mnt/.ix-apps/app_configs/qbittorrent/versions/1.0.10/templates/docker-compose.yaml", line 64, in top-level template code
    }, user=values.qbittorrent.additional_envs, values=values) | tojson }}
^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 485, in getattr
    return getattr(obj, attribute)
           ^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'qbittorren

[EFAULT] Failed to render compose templates: Traceback (most recent call last): File "/usr/bin/apps_render_app", line 33, in sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 47, in main render_templates_from_path(args.path, args.values) File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path rendered_data = render_templates( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 36, in render_templates ).render({'ix_lib': template_libs, 'values': test_values}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render self.environment.handle_exception() File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "/mnt/.ix-apps/app_configs/immich/versions/1.0.5/templates/docker-compose.yaml", line 36, in top-level template code {% if values.immich.enable_ml %} ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 485, in getattr return getattr(obj, attribute) ^^^^^^^^^^^^^^^^^^^^^^^ jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'immich'
remove_circle_outline

More info…

 错误: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 513, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1363, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1360, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 268, in nf
    rv = func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 195, in do_create
    return self.create_internal(job, app_name, version, data['values'], complete_app_details)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 244, in create_internal
    raise e from None
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/crud.py", line 224, in create_internal
    update_app_config(app_name, version, new_values)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 57, in update_app_config
    render_compose_templates(
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/ix_apps/lifecycle.py", line 48, in render_compose_templates
    raise CallError(f'Failed to render compose templates: {cp.stderr}')
middlewared.service_exception.CallError: [EFAULT] Failed to render compose templates: Traceback (most recent call last):
  File "/usr/bin/apps_render_app", line 33, in <module>
    sys.exit(load_entry_point('apps-validation==0.1', 'console_scripts', 'apps_render_app')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 47, in main
    render_templates_from_path(args.path, args.values)
  File "/usr/lib/python3/dist-packages/catalog_templating/scripts/render_compose.py", line 19, in render_templates_from_path
    rendered_data = render_templates(
                    ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/catalog_templating/render.py", line 36, in render_templates
    ).render({'ix_lib': template_libs, 'values': test_values})
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/mnt/.ix-apps/app_configs/immich/versions/1.0.5/templates/docker-compose.yaml", line 36, in top-level template code
    {% if values.immich.enable_ml %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 485, in getattr
    return getattr(obj, attribute)
           ^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'immich'

I’d suggest submitting a bug ticket at this point with a full description of which apps you were attempting to install and how. There is a Report a Bug link if you scroll to the top of this page. After you submit an automatic comment will link you to the private upload service to attach a debug file from the system.