Enable GPU Legacy Drivers

Feature Requests
,
1

Hello,

Below I write up my suggestion to enable legacy drivers support for legacy GPUs. I list what benefits, advantages and disadvantages I can foresee and present a User Story of a potential user, what error they may encounter and how they would resolve it with the requested feature.

Problem/Justification
Many users, including myself, are using older but still fully functional NVIDIA GPUs that are no longer supported by the default drivers in TrueNAS. For example, my Quadro K4000 requires legacy NVIDIA drivers, specifically version 470. Currently, TrueNAS does not provide an option to select or change the NVIDIA driver being used, which results in compatibility issues. This leads to frequent errors, such as k3s pods crashing and restarting constantly, which hinders the system’s usability. An option to select legacy drivers would allow users to continue leveraging their older hardware without sacrificing system stability.

Impact
This feature would benefit a number of TrueNAS users who are running older GPUs that remain capable of handling their workloads. By allowing users to select or change the NVIDIA driver, TrueNAS could accommodate a broader range of hardware, extending the useful lifespan of legacy GPUs. This would improve system reliability for those affected and reduce the need for hardware upgrades solely due to software limitations. The downside could be added complexity in driver management, but this could be mitigated by making the feature optional or advanced-user focused.

  • Benefits:

    • Increased hardware compatibility and utilization, allowing users with older NVIDIA GPUs to fully leverage their existing hardware.
    • Improved system stability and performance by resolving k3s pod crashes and errors associated with driver incompatibility.
    • Reduced e-waste by extending the lifespan of functional hardware.
    • Enhanced user experience by eliminating persistent error messages and resource consumption caused by pod instability.

  • Advantages:

    • Provides a flexible solution for users with diverse hardware configurations.
    • Empowers users to optimize their systems for specific workloads.

  • Disadvantages:

    • Increased complexity in driver management.
    • Potential for user error if incorrect drivers are selected. This can be mitigated through clear documentation and warnings.

User Story
Sarah, an advanced TrueNAS user, relies on her TrueNAS system for storing and processing large design files. Her TrueNAS setup includes a Quadro K4000, a still-capable GPU that is not supported with the current/default TrueNAS NVIDIA driver configuration.

She navigates to the TrueNAS web UI and locates a new “GPU Driver Management” section under “System → Advanced Settings”. Here, she discovers a dropdown menu listing available NVIDIA drivers, including legacy versions. After selecting the recommended driver (version 470) for her Quadro K4000 and clicking “Apply,” TrueNAS automatically installs the selected driver.

Sarah can get back to running her designer-specific workloads without interruptions. She celebrates with a well-deserved cup of coffee, happy that she didn’t have to spend money on a new GPU.

==========Edit==========
I’ve gathered a list of similar issues (almost entirely from the old forums) which would be greatly helped if the end users were able to somehow select which driver is best for their hardware.

Thank you in advance, team! Love your work!

REFERENCES:

https://ixsystems.atlassian.net/browse/NAS-125436
https://ixsystems.atlassian.net/browse/NAS-122988

6 Likes
2

Voted, I also think that ultimately this should be for Nvidia/AMD and Intel GPU’s. There are lots of examples of older hardware that is still more than capable of completing required workloads.

2 Likes
3

I agree with you completely! However, for the sake of the feature and the team that would potentially develop it, I would assume it is more digestible if it is delimited to one brand of GPUs. At least to begin with :sweat_smile:

4

Hopefully with the new installable driver model they developed this will be possible

1 Like
5

Just FYI @scyto and @panzerscope , I’ve updated my post with a bunch of links to similar issues people has posted about, spanning years back. All but one of them are from the old forums.

This is all to say that this is not a new issue. Many would be helped immediately and hopefully also many newcomers in the future.

2 Likes
6

I also have this problem, I have an Nvidia GPU that worked fine under jellyfin. I would not want to have to replace a perfectly functional GPU just because of driver versioning.

1 Like
7

My thoughts exactly! Thank you for voting the issue!

It’s great that slowly but surely people are voting, so we can show it is an issue worth addressing!

8

Exactly my problem. I was so happy that I discovered a GeForce GT730 with a PCIe X1 connector that would work with my X9SCM-F MB (for plex transcoding), only to find out that this card would need nvidia legacy drivers v470. So please consider this feature. Thank you!

1 Like
9

Exactly! Thank you, Christian, for yet another example of a real world use case! :+1:

And also Welcome to the TrueNAS Forums!

10

Chiming in here. We have (now) a “pluggable” nvidia driver framework (since they’re not shipped in base kernel like intel/amd drivers are). The whole legacy GPU debacle is kind of unfortunate since it was, ultimately, nvidia that axed the older generation cards from their “stable” drivers. I can tell you that we have a general idea of how we would accomplish this AND it would even allow us to offer nvidia’s “bleeding edge” drivers as well. But take note that that we can’t simultaneously install the legacy driver and the stable driver, for example. We’ll limit it to only 1 at a time.

I can’t promise that this will happen but I can assure you this has been discussed internally so don’t give up hope (yet) :slight_smile:

10 Likes
11

I would love to see legacy driver support since I’m currently using a P2000 with my TrueNAS SCALE system, and according to Phoronix, Maxwell, Pascal, and Volta support will likely soon move to a Legacy driver.
My understanding is that this would break my current setup since the P in P2000 indicates Pascal.

1 Like
12

Count me in among those who would love to see legacy nvidia driver support.

I’ve got a GeForce 760 in my machine that was recently migrated from Core to Scale. One of the things I was looking forward to was hardware accelerated transcoding of videos in a Plex container…

Earlier today, I got the machine swapped to Scale and rebuilt my Plex jail as a kubernetes container only to find out that the video card isn’t recognized and both dmesg and nvidia-smi indicate that I’d need the 470.xx legacy driver.

1 Like
13

Thank you, @csj for taking the time to give us this perspective into what’s been going on internally! It sounds exciting and it feels good (for myself and all the other voters, I believe) to be acknowledged.

It does sound amazing to be able to support the older cards WHILE still using the latest and greatest drivers, however, if it proves to be too cumbersome to juggle these two benefits, I should think that most users would be happy with just having the option of choosing the exact version to run, even if it doesn’t come with the latest features.

Thank you again for the good work you and the whole team do, and the valuable insight you provided!

14

would this allow us to install say non-legacy grid drivers?

without this it is hard to do true vGPU or support truenas in a VM with vGPU

As an exmample i get this for a truenas VM.

[  130.574832] nvidia 0000:08:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=none
[  130.575076] NVRM: The NVIDIA GPU 0000:08:00.0 (PCI ID: 10de:1e30)
               NVRM: installed in this system is not supported by the
               NVRM: NVIDIA 550.127.05 driver release.
               NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products'
               NVRM: in this release's README, available on the operating system
               NVRM: specific graphics driver download page at www.nvidia.com.
[  130.576347] nvidia: probe of 0000:08:00.0 failed with error -1
[  130.576490] NVRM: The NVIDIA probe routine failed for 1 device(s).
[  130.576619] NVRM: None of the NVIDIA devices were initialized.
[  130.577070] nvidia-nvlink: Unregistered Nvlink Core, major device number 239
[  130.944698] nvidia-nvlink: Nvlink Core is being initialized, major device number 239

for a virtual device like this (the 6000 is not a lagacy card AFAIK based on the grid support tables on nvdias site - but those tables are not easy to parse…)

08:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1)

The only reason i have installed truenas in a VM is because i can’t use arbitrary nvidia drivers on the host (i want to use patched drives)

15

This is a tricky ask. The legacy drivers no longer work with kernel 6.12 or newer.

Maybe they can be patched, and maybe some enterprising user out there is already doing said patching - and, how much effort can ix reasonably put into supporting drivers (and I believe these are blobs not full source) that do not work on current kernels out of the box?

1 Like
16

this the feature i actually need

Support NVIDIA GRID drivers for vGPU (Guest/Host) - Feature Requests - TrueNAS Community Forums

17

I don’t see this request, honestly, going anywhere.

Like @yorick has pointed out, Nvidia’s legacy drivers usually get locked behind older kernels. Nvidia users are already at another crossroad today too. Nvidia’s new Open Source drivers will NOT support non pascal based GPUs going forward. This mean, if they stop providing updates to their closed sourced version, they will also stop supporting those non pascal GPUs too.

Patching the kernel to support such legacy hardware might not be something worth spending the effort required to support.

18

Even the FOSS nova driver, which could conceivably be useable in TrueNAS as early as 26.04, starts with the RTX 2000, Turing.

19

Up voted

20

This the most useful Feature for the people who want to build a strong AIO server with Legacy Hardware by spending less on newer hardware,
Users can grab hardware based on there specific needs…

It is kindy requested to the Developers of Truenas to implement this feature in there next update…

Also I do understand the complexity in the management of different drivers in same system as Nvidea doesn’t allow that…

But still if developers add this features (Legacy Nvidea Driver) it would be very useful for many Truenas Users as well as to those people who
Can’t afford modern hardware