25.10.7 Replacing nVidia card rendered apps unworkable

Apps and Virtualization
, , ,
1

I am trying to replace the nVidia card in my TrueNAS box with something newer (and compatible with the 25.10 nVidia driver.) This is on TrueNAS 25.10.7 Goldeye. The only purpose this GPU serves is as a hardware transcoder for Plex.

I was previously using an old quadro with the old driver using this project (github link I’m not allowed to post)

I removed the modified nvidia driver, installed the new card (a 1660) and did the nvidia cycle (where you uncheck the “Install nVidia drivers” box and then re-check it) but the driver did not download. I tried this several times; the “unchecking” of the box worked fine but upon re-checking the box and applying, the apps service would not start. I tried waiting a long time (20-30 minutes) in case for some reason the driver just took a long time to get, but the driver never downloaded & never appeared on disk.

I finally got frustrated by this process and tried this project: (another github link I’m not allowed to post)

This allowed me to download and build an nVidia driver specifically for my system, which worked OK in that nvidia-smi was satisfied there was a card in the system. It also allowed the Plex app config to at least show that an nvidia card was present and allow me to select it, but upon trying to start plex I got an error:

[2026/09/15 21:00:09] (ERROR) app_lifecycle.compose_action():58 - Failed 'up' action for 'plex' app: Container ix-plex-permissions-1 Creating\n Container ix-plex-permissions-1 Created\n Container ix-plex-plex-1 Creating\n Container ix-plex-plex-1 Created\n Container ix-plex-permissions-1 Starting\n Container ix-plex-permissions-1 Started\n Container ix-plex-permissions-1 Waiting\n Container ix-plex-permissions-1 Exited\n Container ix-plex-plex-1 Starting\nError response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]\n

After un-checking the GPU from the Plex config and reloading/re-checking it, I got a different error, which seems to be persistent at this point, as I don’t get any other error:

[2026/09/15 21:10:01] (ERROR) app_lifecycle.compose_action():58 - Failed 'up' action for 'plex' app: Container ix-plex-permissions-1 Creating\n Container ix-plex-permissions-1 Created\n Container ix-plex-plex-1 Creating\n Container ix-plex-plex-1 Created\n Container ix-plex-permissions-1 Starting\nError response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to create the automatic CDI modifier: failed to generate CDI spec for mode "auto": failed to construct device spec generators: failed to get device handle from UUID: Not Found: unknown\n

It’s like some task can’t get the UUID of the device I’m trying to map.

All the while, nvidia-smi is perfectly happy a card is installed:

Sat Sep 19 16:46:22 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.99.02              Driver Version: 595.99.02      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1660 ...    Off |   00000000:01:00.0 Off |                  N/A |
| 25%   39C    P0             39W /  125W |       0MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

nvidia-smi -L
GPU 0: NVIDIA GeForce GTX 1660 SUPER (UUID: GPU-54477341-6eb6-37bb-18ae-a374347813cc)

nvidia-smi -B
No excluded devices found.

nvidia-smi -q

==============NVSMI LOG==============

Timestamp                                              : Sat Sep 19 16:50:16 2026
Driver Version                                         : 595.99.02
CUDA Version                                           : 13.2

Attached GPUs                                          : 1
GPU 00000000:01:00.0
Product Name                                       : NVIDIA GeForce GTX 1660 SUPER
Product Brand                                      : GeForce
Product Architecture                               : Turing
Display Mode                                       : Requested functionality has been deprecated
Display Attached                                   : No
Display Active                                     : Disabled
Persistence Mode                                   : Disabled
Addressing Mode                                    : None
MIG Mode
Current                                        : N/A
Pending                                        : N/A
Accounting Mode                                    : Disabled
Accounting Mode Buffer Size                        : 4000
Driver Model
Current                                        : N/A
Pending                                        : N/A
Serial Number                                      : N/A
GPU UUID                                           : GPU-54477341-6eb6-37bb-18ae-a374347813cc
GPU PDI                                            : 0x2a914de791558043
(SNIP)

I am REALLY super lost about where to even go from here, thinking at this point about either trying to go back to the old dated Quadro with the old dated driver, which at least worked, or doing a system reinstall of some kind (but really not wanting to recreate literally everything just to get this GPU working.) I very much would like to go back to the system, unadulterated driver as that was the whole point of replacing my GPU.

Any suggestions?

2

yeah that CDI line “failed to get device handle from UUID: Not Found” usually means the apps/docker nvidia runtime is out of sync with the hand-built driver, even when nvidia-smi looks fine. plex is asking for the nvidia device path and the auto CDI generator cant see that UUID.

stuff that usually gets me back toward stock on 25.10:

  1. strip the custom nvidia build so you are not mixing trees with the official apps toggle
  2. stop apps, reboot once with the new 1660 in
  3. use the official Install nVidia drivers checkbox again and watch journalctl -u middlewared while it applies. if the driver never lands on disk, it is often a download/mirror or boot-pool space issue, not the card
  4. after stock driver is actually installed, check that CDI/nvidia-container bits can see GPU-54477341-6eb6-37bb-18ae-a374347813cc (nvidia-smi -L UUID). the “could not select device driver nvidia” / UUID Not Found pair is classic when smi works but the container runtime never registered the toolkit

if you want the old Quadro path as a temporary escape hatch that is fine, but for the 1660 the goal is really stock SCALE nvidia + clean CDI, not another custom tree. paste middlewared lines from the failed nvidia checkbox apply if it still wont download.

3

Thanks for your help nite_route, here’s what I tried (though I’m ultimately still stuck)

  1. Disabled “install nvidia drivers,” then removed custom driver from disk and rebooted.
  2. Waited for system to stabilize, started journalctl -fu middlewared
  3. Checked the ‘install nvidia drivers’ box
  4. System stayed at ‘app service not running’ status for quite a while. Waited ~10 minutes but no nvidia related entries ever appeared in middlewared log.
  5. Did the uncheck->recheck cycle again because if the printer’s not working why not try sending the job two or three more times, right? lolz
  6. Same result as above

The driver never appeared to re-download, so it never worked. That’s kind of where I’m stuck with the stock driver, it just never downloads.

I was able to get it working this way, though that’s not the way I want it to work:

  1. Use the custom driver installer to load the driver file.
  2. Recheck ‘install nvidia driver’
  3. Use the midclt job listed here to manually update the plex app with the right slot ID and UUID for the GPU Docker Apps and UUID issue with NVIDIA GPU after upgrading TrueNAS
  4. Plex starts and uses the GPU

I would still really like to get back to the stock driver so that I don’t have to worry about upgrades anymore but at least something is working.

Thanks again for your help.

4

yeah if middlewared never even logs a nvidia download attempt that usually means the job never started, not that the mirror is slow. a few things id check next on 25.10:

df -h / (boot-pool space) and make sure it is not near full
System Settings → Network, confirm the box can reach the internet without a weird proxy
from the shell try refreshing catalogs (midclt call catalog.sync or whatever your build exposes) and see if catalogs refresh at all. if catalog sync is broken the nvidia package never gets queued either
ls wherever the stock driver lands on your build after a failed apply, just to confirm it is still empty
also after stripping the custom tree, make sure nothing leftover is still pointing nvidia-container / CDI at the hand-built path

if catalogs refresh fine but the checkbox still never logs, paste the exact UI toast or job id from Tasks. the midclt UUID hack keeping Plex alive is fine as a bridge, just means you are still on the custom driver underneath.

5

Thanks again. The boot pool is definitely not full, it’s on a 1TB SSD and nothing else is on it.

I desperately wish this was a networking problem, THAT I could fix. I really don’t think it is; midclt call catalog.sync works fine:

{
"id": 10950,
"message_ids": [
"a4e64913-33ef-45b4-9b36-7238c808f58a"
],
"method": "catalog.sync",
"arguments": [],
"transient": false,
"description": null,
"abortable": false,
"logs_path": null,
"logs_excerpt": null,
"progress": {
"percent": 100,
"description": "Synced 'TRUENAS' catalog",
"extra": null
},
"result": null,
"result_encoding_error": null,
"error": null,
"exception": null,
"exc_info": null,
"state": "SUCCESS",
"time_started": {
"$date": 1790222495000
},
"time_finished": {
"$date": 1790222498000
},
"credentials": null
}
]

I did some more investigation about what jobs have happened when the checkbox is checked or unchecked, but no errors are being reported there. I tried rebooting the system with the box unchecked and then re-checking the box, but even at that point no job starts that seems to do any downloading.

What is interesting is when the box is checked, a link appears in /run/extensions to nvidia.raw, even if it’s a link to nothing. but at no point does /usr/share/truenas/sysext-extensions/nvidia.raw appear, no job appears to download it, no errors appear that I can find.

I’m pretty lost here.

6

gotcha. catalog.sync working plus a dead symlink at /run/extensions/nvidia.raw and nothing under /usr/share/truenas/sysext-extensions/ means the checkbox is arming the extension path but the nvidia.raw fetch job never actually runs.

next checks on 25.10.7:

midclt call core.get_jobs ‘[[“method”, “rin”, “nvidia”]]’ | jq
midclt call system.advanced.config | jq
ls -la /run/extensions /usr/share/truenas/sysext-extensions 2>/dev/null
find /var/cache /var/db /var/lib -name ‘nvidia.raw’ 2>/dev/null | head

also from shell after checking the box:
journalctl -u middlewared -b --no-pager | rg -i ‘nvidia|sysext|extension’ | tail -50

if get_jobs shows nothing nvidia related at all when you flip the box, the UI toggle is not enqueueing nvidia.update / nvidia.install on your build. paste that get_jobs output and whatever journalctl shows right after a fresh check+Apply. the custom driver UUID hack keeping plex up is fine for now, just means stock never registered.