HOWTO: Keylase Nvidia Patch

TJett · January 10, 2025, 12:29am

The following is when I run it manually.

root@truenas[]# ./patch.sh 
Already up to date.
mount: /usr: cannot remount sysext read-write, is write-protected.
       dmesg(1) may have more information after failed mount system call.

I don’t see any logs from the post-init script, but I’m assuming it’s running into the same issue.

You have it working without do anything with the unmerge/merge that was mentioned a few comments up? The way I’m actually testing it is opening my tdarr docker container that has ffmpeg and running 9 dummy encodes, and when it fails on that 9th on I know it didn’t work. Let me know if I should be testing it differently though lol.

ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda \
      -f lavfi -i testsrc -t 50 \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 1M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 2M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 3M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 4M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 5M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 6M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 7M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 8M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 9M -f null -

…

[h264_nvenc @ 0x5575163385c0] OpenEncodeSessionEx failed: incompatible client key (21): (no details)
[vost#8:0/h264_nvenc @ 0x5575163382c0] Error initializing output stream: Error while opening encoder for output stream #8:0 - maybe incorrect parameters such as bit_rate, rate, width or height

dasunsrule32 · January 10, 2025, 12:30am

I’ll have to take a look. It’s working on my box, but I upgraded all the way back from Cobia and the releases between EE.

sean6541 · January 19, 2025, 1:54am

Here’s what I’ve done on 24.10:

Create a folder in the root of your app dataset (anywhere works technically). Mine is /mnt/System/nvidia-patch
Put the patch.sh from Keylase GitHub in the folder your created.
Copy and paste the following into a new file run.sh in the same folder, replacing the </path/to/folder/from/step/1> with the path to the folder you created in step 1:

#!/bin/sh

PATCH_DIR="</path/to/folder/from/step/1>"

sed -i -r "s,^backup_path=.*,backup_path=\"$PATCH_DIR/libnvidia-encode-backup\",g" "$PATCH_DIR/patch.sh" && \
if [ -d "$PATCH_DIR/patched-lib" ]; then rm -r "$PATCH_DIR/patched-lib"; fi && \
mkdir -p "$PATCH_DIR/patched-lib" && \
PATCH_OUTPUT_DIR="$PATCH_DIR/patched-lib" "$PATCH_DIR/patch.sh" && \
cd "$PATCH_DIR/patched-lib" && \
for f in * ; do
    suffix="${f##*.so}"
    name="$(basename "$f" "$suffix")"
    mount --bind "$f" "/usr/lib/x86_64-linux-gnu/$f"
    mount --bind "$f" "/usr/lib/x86_64-linux-gnu/$name"
    mount --bind "$f" "/usr/lib/x86_64-linux-gnu/$name.1"
done && \
ldconfig

Add the run.sh script you just created as a pre-init script in the WebUI.
Reboot and done!

mew · January 23, 2025, 5:32pm

Thank you so much! This worked flawlessly

dasunsrule32 · January 29, 2025, 7:32pm

I checked on my system and indeed it is no longer working on 24.10.1+. I had tested this on 24.10.0 and it was working fine there and mistakenly spoke that it was still working. I apologize for that mistake.

Since 24.10.1, IX has moved to systemd-sysext for the nvidia driver. This can be seen by running the following commands with the following output:

systemd-sysext status
HIERARCHY EXTENSIONS SINCE                      
/opt      none       -                          
/usr      nvidia     Wed 2025-01-29 09:30:26 EST

systemd-sysext list 
NAME   TYPE PATH                       TIME                       
nvidia raw  /run/extensions/nvidia.raw Mon 2025-01-27 23:20:11 EST

It’s pulling in what looks like a custom built nvidia image, /usr/share/truenas/sysext-extensions/nvidia.raw and merging that into /usr. I’m assuming it’s been packaged by IX and downloaded when triggering the Install Nvidia Drivers option in the WebUI using the middleware. As a result, it creates a temporary readonly status on the zfs dataset, which in turn doesn’t allow remounting to writable in newer versions of TNS. This is the way it will be moving forward as it has already been backported to TNS 25.04+.

zfs get readonly boot-pool/ROOT/24.10.2/usr
NAME                        PROPERTY  VALUE   SOURCE
boot-pool/ROOT/24.10.2/usr  readonly  on      temporary

The proper way to do this would be one of two ways:

Create a custom packed sysext image with the modified binaries.
Mounting --bind like @sean6541 has done.

If @sean6541 is ok with it, I’ll modify my post with his script slightly modified to pull in the updated repo so the latest patch.sh will always be in use.

dasunsrule32 · January 29, 2025, 9:14pm

Ok, a slight rewrite of @sean6541 script in bash.

Create a dataset for the script and run it from there. Set it to run POSTINIT and you should be good. I removed the other mount commands because those are symlinks.

This version also fixes an issue where the original script would mount multiple lib files.

mount|grep libnvidia                    
pool/data/scripts on /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05 type zfs (rw,relatime,xattr,nfs4acl,casesensitive)
pool/data/scripts on /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05 type zfs (rw,relatime,xattr,nfs4acl,casesensitive)
pool/data/scripts on /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05 type zfs (rw,relatime,xattr,nfs4acl,casesensitive)
pool/data/scripts on /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05 type zfs (rw,relatime,xattr,nfs4acl,casesensitive)

Script:

#!/bin/bash

set -x

PATCH_DIR="$PWD"
KEYLASE_GH="https://raw.githubusercontent.com/keylase/nvidia-patch/refs/heads/master/patch.sh"
KEYLASE_SCRIPT="$PATCH_DIR/patch.sh"

# Download latest keylase script
wget -O $KEYLASE_SCRIPT $KEYLASE_GH && \

# Update backup_path
sed -i -r "s,^backup_path=.*,backup_path=\"$PATCH_DIR/libnvidia-encode-backup\",g" "$KEYLASE_SCRIPT" && \

# Add output path if it doesn't exist
[[ ! -f $PATCH_DIR/patched-lib ]] && mkdir -p "$PATCH_DIR/patched-lib"

# Patch libs
PATCH_OUTPUT_DIR="$PATCH_DIR/patched-lib" bash "$KEYLASE_SCRIPT" && \

# Get current mounted nvidia libs
NV_MOUNTED_LIBS=`awk '/libnvidia/ { print $2 }' < /proc/mounts`

cd "$PATCH_DIR/patched-lib" && \

# Mount patched libs
for file in * ; do
   suffix="${file##*.so}"
   name="$(basename "$file" "$suffix")"

   # Check if old libraries are mounted
   # and unmount if so, else continue
   # mounting.
   if [[ ! -z $NV_MOUNTED_LIBS ]]; then
      i=0
      for NV_LIB in ${NV_MOUNTED_LIBS[$i]} ; do
         ((i++))
         umount $NV_LIB
         NV_LIB_BASE=`basename $NV_LIB`
         mount --bind "$NV_LIB_BASE" "$NV_LIB"
      done
   else
      mount --bind "$file" "/usr/lib/x86_64-linux-gnu/$file"
   fi
done && \

# Link libs
ldconfig

Output:

./ts-nvidia-patch.sh    
+ PATCH_DIR=/mnt/pool/data/scripts/nvidia
+ KEYLASE_GH=https://raw.githubusercontent.com/keylase/nvidia-patch/refs/heads/master/patch.sh
+ KEYLASE_SCRIPT=/mnt/pool/data/scripts/nvidia/patch.sh
+ wget -O /mnt/pool/data/scripts/nvidia/patch.sh https://raw.githubusercontent.com/keylase/nvidia-patch/refs/heads/master/patch.sh
--2025-01-29 19:31:23--  https://raw.githubusercontent.com/keylase/nvidia-patch/refs/heads/master/patch.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28264 (28K) [text/plain]
Saving to: ‘/mnt/pool/data/scripts/nvidia/patch.sh’

/mnt/pool/data/scripts/nvidia/patch 100%[================================================================>]  27.60K  --.-KB/s    in 0.003s  

2025-01-29 19:31:23 (8.90 MB/s) - ‘/mnt/pool/data/scripts/nvidia/patch.sh’ saved [28264/28264]

+ sed -i -r 's,^backup_path=.*,backup_path="/mnt/pool/data/scripts/nvidia/libnvidia-encode-backup",g' /mnt/pool/data/scripts/nvidia/patch.sh
+ [[ ! -f /mnt/pool/data/scripts/nvidia/patched-lib ]]
+ mkdir -p /mnt/pool/data/scripts/nvidia/patched-lib
+ PATCH_OUTPUT_DIR=/mnt/pool/data/scripts/nvidia/patched-lib
+ bash /mnt/pool/data/scripts/nvidia/patch.sh
Detected nvidia driver version: 550.127.05
libnvidia-encode.so
Backup exists and driver file differ from backup. Skipping patch.
++ cat /proc/mounts
++ grep libnvidia
++ awk '{print($2)}'
+ NV_MOUNTED_LIBS='/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05
/mnt/pool/data/scripts/nvidia/patched-lib/libnvidia-encode.so.550.127.05'
+ cd /mnt/pool/data/scripts/nvidia/patched-lib
+ for file in *
+ suffix=.550.127.05
++ basename libnvidia-encode.so.550.127.05 .550.127.05
+ name=libnvidia-encode.so
+ [[ ! -z /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05
/mnt/pool/data/scripts/nvidia/patched-lib/libnvidia-encode.so.550.127.05 ]]
+ for NV_LIB in ${NV_MOUNTED_LIBS[@]}
+ umount /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05
++ basename /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05
+ NV_LIB_BASE=libnvidia-encode.so.550.127.05
+ mount --bind libnvidia-encode.so.550.127.05 /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05
+ ldconfig

Mount output:

mount|grep libnvidia       
pool/data/scripts on /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05 type zfs (rw,relatime,xattr,nfs4acl,casesensitive)

Files output:

ll /usr/lib/x86_64-linux-gnu/libnvidia-encode*
lrwxrwxrwx 1 root root   21 Jan 27 23:19 /usr/lib/x86_64-linux-gnu/libnvidia-encode.so -> libnvidia-encode.so.1
lrwxrwxrwx 1 root root   30 Jan 27 23:19 /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1 -> libnvidia-encode.so.550.127.05
-rw-r--r-- 1 root root 271K Jan 29 16:09 /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.127.05

dasunsrule32 · January 30, 2025, 4:38pm

Updated OP.