LocalAI app GPU usage Nvidia driver issue Tesla P40

[dnb08]

Howdy all,

I’m trying to setup the LocalAI app and use my GPU but it doesn’t seem to be well documented. Has anyone got this working, if so how? does anyone know where the app setup is documented?

[LarsR]

how do you deploy the app? custom app via compose or a truenas app?

[dnb08]

I am trying to get the localAI app to use my GPU. I have apps installed and models loaded but it isnt using the gpu

[dnb08]

seems to be related NVIDIA GPUs not working and nvidia-smi fails to communicate with NVIDIA drivers - #62 by kuyper

[SmallBarky]

It would really help if you posted your detailed hardware and OS version. There were NVIDIA driver changes with 25.10 and we don’t know if you are on old or new GPUs

[dnb08]

looks like the change to the open drive is the issue as i am on a Tesla p40 NVIDIA 570.172.08 driver. supports it but not the open version. It would be great to support some of these cards as they are cheep and lots of vram. I’m on 25.10.4 - Goldeye. Anyone know a work around?

[SmallBarky]

Try this thread. There is a work around to install the older driver. Link puts you near end of the thread

NVIDIA Kernel Module Change in TrueNAS 25.10 - What This Means for You

Mind dropping your TrueNAS 25.10.1 compiled 580.105.08 driver nvidia.raw file on github?

There is also this thread where the even the older drivers are dropping support for those cards, I think.

TrueNAS 26.0.0-BETA.1 NVIDIA 590.44.01 no longer supports Tesla P4 (10de:1bb3)

The last drivers to support the P4 will be the 580-series drivers, so anything in that vein (580.190.04 seems to be the Very Latest of them on the Linux/Unix datacenter side?) Regardless of open vs proprietary module source, it was NVIDIA who deprecated and then cut off the Pascal/Volta cards in the 590 driver line as @DjP-iX points out - they will not work even with the proprietary kmod compiled. I have a couple of P4s that were impacted by this, so I share your frustration, especially since …

[dnb08]

not at all just tell me how to get what you need. New to truenas

[dnb08]

Running the script I get the following

sudo curl -sSL https://raw.githubusercontent.com/zzzhouuu/truenas-nvidia-drivers/refs/heads/main/scripts/install.sh | bash
Detected TrueNAS version: 25.10.4
Downloading NVIDIA drivers 25.10.4…
/tmp/nvidia.raw 100%[=================================================>] 469.25M 109MB/s in 4.5s
/tmp/nvidia.raw.sha256 100%[=================================================>] 65 --.-KB/s in 0s
Verifying SHA256 checksum…
SHA256 checksum verification passed.
Disabling NVIDIA support…
Need to be privileged.
truenas_admin@truenas[~]$

[SmallBarky]

How are you running the script. I think it needs to be run as ROOT. @zzzhouuu can you give feedback?

[dnb08]

Fixed it after I thought for a sec

curl -sSL https://raw.githubusercontent.com/zzzhouuu/truenas-nvidia-drivers/refs/heads/main/scripts/install.sh | sudo bash

[dnb08]

truenas_admin@truenas[~]$ curl -sSL https://raw.githubusercontent.com/zzzhouuu/truenas-nvidia-drivers/refs/heads/main/scripts/install.sh | sudo bash
Detected TrueNAS version: 25.10.4
Downloading NVIDIA drivers 25.10.4…
/tmp/nvidia.raw 100%[=================================================>] 469.25M 118MB/s in 4.0s
/tmp/nvidia.raw.sha256 100%[=================================================>] 65 --.-KB/s in 0s
Verifying SHA256 checksum…
SHA256 checksum verification passed.
Disabling NVIDIA support…
Installing NVIDIA drivers…
Enabling NVIDIA support…
No extensions found.
NVIDIA drivers installed successfully.
truenas_admin@truenas[~]$
truenas_admin@truenas[~]$ nvidia-smi

No devices were found
truenas_admin@truenas[~]$ sudo nvidia-smi

No devices were found
truenas_admin@truenas[~]$ lsmod | grep nvidia

nvidia_uvm 2019328 0
nvidia_drm 135168 0
nvidia_modeset 1830912 1 nvidia_drm
nvidia 97435648 2 nvidia_uvm,nvidia_modeset
drm_ttm_helper 16384 1 nvidia_drm
drm_kms_helper 253952 4 drm_display_helper,drm_ttm_helper,nvidia_drm,i915
drm 770048 9 drm_kms_helper,drm_display_helper,nvidia,drm_buddy,drm_ttm_helper,nvidia_drm,i915,ttm
video 81920 2 i915,nvidia_modeset
truenas_admin@truenas[~]$ lspci | grep -i nvidia

01:00.0 3D controller: NVIDIA Corporation GP102GL [Tesla P40] (rev a1)
truenas_admin@truenas[~]$ sudo nvidia-smi -pm 1
No devices were found
truenas_admin@truenas[~]$

[dnb08]

fixed the driver issue by enabling G4 in the bios.

sudo nvidia-smi
Sat Jun 27 06:16:32 2026
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.8 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P40 On | 00000000:01:00.0 Off | 0 |
| N/A 25C P0 48W / 250W | 0MiB / 23040MiB | 0% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+

±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
±----------------------------------------------------------------------------------------+

I can now pass the card to the localAI app but it doesnt seem to be using it. deleted and reinstalled the app re downloading models to see if that works