Electric Eel with Jailmaker and nVidia

Okay, need some help here from the community. I researched and found that Electric Eel supports jailmaker and jails would move over however it seems that my Plex jail is complaining that the nVidia drivers are no longer included by default as expected according to the release notes. Has anyone gotten around this? I’ve tried going into the new apps configuration and checking the box for the Nvidia drivers. It says it’s installed and there despite getting an error about them not being compiled correctly. Has anyone gotten jailmaker jails to work with nVidia pass through again after upgrading to Electric Eel?

Install error:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 488, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 535, in __run_body
rv = await self.middleware.run_in_thread(self.method, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1364, in run_in_thread
return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1361, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/nvidia.py”, line 65, in install
self._install_driver(job, td, path)
File “/usr/lib/python3/dist-packages/middlewared/plugins/nvidia.py”, line 133, in _install_driver
subprocess.run([path, “–tmpdir”, td, “-s”], capture_output=True, check=True, text=True)
File “/usr/lib/python3.11/subprocess.py”, line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command ‘[’/root/tmpbwr1mlcn/NVIDIA-Linux-x86_64-550.127.05-no-compat32.run’, ‘–tmpdir’, ‘/root/tmpbwr1mlcn’, ‘-s’]’ returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 488, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 533, in __run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 49, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 179, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/docker/update.py”, line 106, in do_update
await (
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 436, in wait
raise self.exc_info[1]
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 492, in run
raise handled
middlewared.service_exception.CallError: [EFAULT] Command /root/tmpbwr1mlcn/NVIDIA-Linux-x86_64-550.127.05-no-compat32.run --tmpdir /root/tmpbwr1mlcn -s failed (code 1):
Verifying archive integrity… OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 550.127.05…

ERROR: Unable to load the kernel module ‘nvidia.ko’. This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or if another driver, such as nouveau, is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA device(s), or no NVIDIA device installed in this system is supported by this NVIDIA Linux graphics driver release.

Please see the log entries ‘Kernel module load error’ and ‘Kernel messages’ at the end of the file ‘/var/log/nvidia-installer.log’ for more information.

ERROR: Installation has failed. Please see the file ‘/var/log/nvidia-installer.log’ for details. You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.

What does the output of nvidia-smi -L show, and have you attempted to isolate any GPUs through the interface?

When I access the root shell on the system and try and run that command this is what I get. My nVidia card shows up in the list to isolate however I currently don’t have any of them isolated as I read that is for VMs and not jails.

root@nas01[~]# nvidia-smi -L
zsh: command not found: nvidia-smi

image

Okay, it definitely didn’t install the NVIDIA drivers then, because you’re already root - and if you haven’t isolated it there should be no reason for it to have been claimed by vfio.

Can you attach the /var/log/nvidia-installer.log file to the thread, or post the contents in codeblocks?

I have a P4 as well so let me see if I can reproduce this.

@HoneyBadger here is the log file you requested.
nvidia-installer.txt (892.4 KB)

Turns out my Tesla system is still on RC2, I’ll have to upgrade it. Stand by a second. :slight_smile:

But the log snippet shows a successful module build followed by an inability to load it.

[ 3884.286927] NVRM: The NVIDIA probe routine was not called for 1 device(s).
[ 3884.288364] NVRM: This can occur when another driver was loaded and 
               NVRM: obtained ownership of the NVIDIA device(s).
[ 3884.288367] NVRM: Try unloading the conflicting kernel module (and/or
               NVRM: reconfigure your kernel without the conflicting
               NVRM: driver(s)), then try loading the NVIDIA kernel module
               NVRM: again.

NVIDIA driver build ignored my legacy (boot/console) GPU and proceeded to greenlight the P4.

[  178.754318] nvidia-nvlink: Nvlink Core is being initialized, major device number 235
[  178.754325] NVRM: The NVIDIA GeForce GT 710 GPU installed in this system is
               NVRM:  supported through the NVIDIA 470.xx Legacy drivers. Please
               NVRM:  visit http://www.nvidia.com/object/unix.html for more
               NVRM:  information.  The 550.127.05 NVIDIA driver will ignore
               NVRM:  this GPU.  Continuing probe...
[  178.755926] NVRM: ignoring the legacy GPU 0000:02:00.0
[  178.755933] nvidia: probe of 0000:02:00.0 failed with error -1
[  178.756006] nvidia 0000:81:00.0: enabling device (0000 -> 0002)
[  178.872030] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  550.127.05  Tue Oct  8 03:22:07 UTC 2024
[  178.945589] nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
[  179.005827] nvidia-uvm: Loaded the UVM driver, major device number 511.
[  179.023643] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  550.127.05  Tue Oct  8 02:56:05 UTC 2024
[  179.032480] [drm] [nvidia-drm] [GPU ID 0x00008100] Loading driver
[  179.032483] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:81:00.0 on minor 1
[  179.039853] [drm] [nvidia-drm] [GPU ID 0x00008100] Unloading driver
[  179.081867] nvidia-modeset: Unloading
[  179.131738] nvidia-uvm: Unloaded the UVM driver.
[  179.182934] nvidia-nvlink: Unregistered Nvlink Core, major device number 235

I don’t use Plex, but I’ll see if I can make a jlmkr instance of Jellyfin.

root@supermicro[/var/log]# nvidia-smi
Thu Oct 31 08:26:37 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P4                       Off |   00000000:81:00.0 Off |                  Off |
| N/A   34C    P0             22W /   75W |       0MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Could this be caused by an upgrade from SCALE to EE? I was working fine on SCALE and just did the upgrade to EE.

@HoneyBadger I think I figured out the issue.

After reading your posts I got to thinking that it was Jailmaker that was grabbing the nVidia card and preventing the install of the driver. I configured the Plex jail to NOT use nVidia and restarted the jail. After that I then restarted the server to clear everything out. After the restart I went in to Apps → Configure → Settings and checked the box and low and behold it installed just fine as shown below from the kernal log and the nvidia-smi app. I then reconfigured the Plex jail to allow the nVidia pass in and restarted it. However at this time it appears to give a new error that it can’t find the nvidia-current-nvm in /lib/modules/6.6.44-production+truenas but the jails starts now.

Blockquote
Oct 31 12:54:08 nas01…com kernel: VFIO - User Level meta-driver version: 0.3
Oct 31 12:54:09 nas01…com kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 238
Oct 31 12:54:09 nas01…com kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module 550.127.05 Tue Oct 8 03:22:07 UTC 2024
Oct 31 12:54:09 nas01…com kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
Oct 31 12:54:09 nas01…com kernel: nvidia-uvm: Loaded the UVM driver, major device number 236.
Oct 31 12:54:09 nas01…com kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 550.127.05 Tue Oct 8 02:56:05 UTC 2024
Oct 31 12:54:09 nas01…com kernel: [drm] [nvidia-drm] [GPU ID 0x00000700] Loading driver
Oct 31 12:54:09 nas01…com kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:07:00.0 on minor 1
Oct 31 12:54:09 nas01…com kernel: [drm] [nvidia-drm] [GPU ID 0x00000700] Unloading driver
Oct 31 12:54:09 nas01…com kernel: nvidia-modeset: Unloading
Oct 31 12:54:09 nas01…com kernel: nvidia-uvm: Unloaded the UVM driver.
Oct 31 12:54:09 nas01…com kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 238
Oct 31 12:54:43 nas01…com kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 239
Oct 31 12:54:43 nas01…com kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module 550.127.05 Tue Oct 8 03:22:07 UTC 2024
Oct 31 12:54:43 nas01…com kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 550.127.05 Tue Oct 8 02:56:05 UTC 2024
Oct 31 12:54:43 nas01…com kernel: [drm] [nvidia-drm] [GPU ID 0x00000700] Loading driver
Oct 31 12:54:43 nas01…com kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:07:00.0 on minor 1
^C

Blockquote
root@nas01[/var/log]# nvidia-smi
Thu Oct 31 12:55:25 2024
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05 Driver Version: 550.127.05 CUDA Version: 12.4 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla P4 Off | 00000000:07:00.0 Off | 0 |
| N/A 48C P0 23W / 75W | 0MiB / 7680MiB | 2% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
±----------------------------------------------------------------------------------------+

I have confirmed that Plex is doing hardware transcoding so I believe that it is seeing the GPU.

image

Also verified with nvidia-smi.