NVIDIA Kernel Module Change in TrueNAS 25.10 - What This Means for You

Announcements
,
122

well now i feel foolish, the nvidia documentation clearly states one needs >=575.x to enable MIG, turns out that is not true…..

truenas_admin@truenas1 ~ 22:44:11 $ sudo nvidia-smi
Sat Mar  7 22:44:31 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:0F:00.0 Off |                  Off |
| 37%   36C    P0            152W /  600W |     256MiB /  97887MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                     Memory-Usage |        Vol|        Shared         |
|      ID  ID  Dev |                       BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  0    3   0   0  |              64MiB / 24192MiB    | 46    N/A |  1   1    1    0    1 |
|                  |                 0MiB / 32767MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    4   0   1  |              64MiB / 24192MiB    | 46    N/A |  1   1    1    0    1 |
|                  |                 0MiB / 32767MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    5   0   2  |              64MiB / 24192MiB    | 46    N/A |  1   1    1    0    1 |
|                  |                 0MiB / 32767MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    6   0   3  |              64MiB / 24192MiB    | 46    N/A |  1   1    1    0    1 |
|                  |                 0MiB / 32767MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

tl;dr the built in scale 25.10.2.1 drivers allow MIG to be enabled
leaving only minor issues:

  1. no persistence daemon still
  2. no displaymodeswitch or easy way to install that / run that
  3. no built in solution for a)creating MIGs at boot and b)assigning them to containers - but guess this might be better for a seperate topic…..
  4. tl;dr my work will still offer some benefits….. and i now know how to create sysexts for anything i want…..
123

we’re building 590.x in there actually, you can check in the nightlies now if you like :slight_smile:

  1. Should be there, just not on by default. Post-init task sudo nvidia-smi -pm 1 should do it globally, or you can target individual cards by slot/uuid/enumeration.
  2. Nothing offhand for this one, sorry.
  3. We can’t distribute the driver that enables MIGs and vGPU.
  4. Being able to package custom sysexts for TrueNAS certainly has potential to extend TrueNAS capabilities into more community focused areas without being reliant on iXsystems :wink:
1 Like
124

sure, good enough woraround for most, but when they run that the user will get an error it is deprecated, also on my card i got lower W usage with the new driver / persistenced (not sure if it is the driver or the persistenced)

125

woo hoo, thats great :slight_smile:

126

Might be the driver improvements considering you’re on some bleeding edge gear there. (Workstation, Server, or Max-Q?)

nvidia-persistenced was also occasionally getting in the way of being able to do things like dynamically unload the sysext module or unbind the driver to vfio a card - and worst of all it was inconsistent, which is the bestest kind of bug to fix because it doesn’t reliably reproduce.

1 Like
127

just FYI the Workstation card uses the same open driver you already distribute, it enables MIG, no license is needed for that card in MIG mode (this is different to the server cards) already got your 570 driver working with MIG :slight_smile:

agree about the vGPU or anything needing a license,

1 Like
129

Mind dropping your TrueNAS 25.10.1 compiled 580.105.08 driver nvidia.raw file on github?

1 Like
LocalAI app GPU usage Nvidia driver issue Tesla P40
130
131

Wanted to flag a different but related issue for anyone hitting GPU trouble on 25.10: Thunderbolt-connected eGPUs appear to have a genuine compatibility gap with the open kernel driver, separate from the MIG/Blackwell stuff above.

My setup: RTX 3070 (GA104) in a Razer Core X V2, connected via Thunderbolt to a TrueNAS SCALE 25.10 host. Summary of what I found:

  • The Thunderbolt device isn’t auto-authorized — needed manual echo 1 > /sys/bus/thunderbolt/devices/X-X/authorized before the GPU even appears in lspci

  • First driver load attempts consistently hit Xid 79 (“GPU has fallen off the bus”) during the GSP firmware RPC handshake (kgspWaitForRmInitDone, kernel_gsp_tu102.c:533)

  • Ruled out: PCIe BAR allocation (properly assigned), power state (confirmed D0, not D3cold), ASPM policy (tested performance mode, no change)

  • Setting iommu=pt cleared some DMAR PTE faults I was seeing on the GPU’s PCI address, but the underlying issue persisted

  • On a subsequent attempt post-IOMMU-passthrough, hit an actual kernel crash inside the driver’s PCIe capability parser — clSetPortPcieCapOffset_IMPL → objClInitGpuPortData → clInitPcie_IMPL → RmInitAdapter, wedging the driver’s persistent _main_loop kthread and leaving nvidia-smi as a zombie process

Given NVIDIA’s own note that GeForce/Workstation support in the open kernel modules is alpha-quality, my read is this is a real edge case in how the driver parses PCIe capabilities specifically over a Thunderbolt-tunneled topology — likely never tested against that combination.

Happy to provide full dmesg logs if useful for triage. Curious if anyone else running a consumer GeForce card through a Thunderbolt enclosure (rather than a native PCIe slot) has hit this same wall.

132

I think you would be suited to running on a regular Linux distribution and submitting feedback or bug reports for your setup there. I don’t see TrueNAS devs working on Thunderbolt related issues unless it would affect Enterprise customers.

2 Likes
133

I know we’ve got a couple folks using some of that, but “eGPU over ThunderBolt” is definitely something that’s outside our usual wheelhouse as mentioned by @SmallBarky

NVIDIA is tracking a few bugs upstream in their internal system w.r.t the GSP/Open firmware, they’ve squashed a few in the later releases including the failure to init on RTX 5060(Ti) cards, so it might be a factor of trying to manually build the newer drivers against the 25.10 kernel and see if it’s been fixed in 590 or 610 even - if you’re interested in that kind of dev shenaigans.

1 Like