Nvidia container toolkit

Feature Requests
1

Problem/Justification
The Nvidia Container Toolkit is “a collection of libraries and utilities enabling users to build and run GPU-accelerated containers”. At the moment, Nvidia based hardware is not fully available for trueNAS users, unlike Intel GPUs whose toolkit has already been enabled in TrueNAS.

Impact
Any TN user wishing to use docker containers will be disadvantaged without the toolkit if their container needs to use Nvidia GPU hardware. For example, Frigate users are unable to implement more advanced configurations of Frigate because the TN provided app has limitations and it is difficult to know how the app implements Frigate. Furthermore, the Frigate GitHub community provides full support for Docker methods but none for TrueNAS app methods.

User Story
Frigate is a superbly written network video recording application which is able to employ GPU acceleration to provide machine learning and AI functionality.
The TrueNAS app is one way to implement it but the other way, preferred in some cases due to the extra flexibility it would enable, is to use docker compose (perhaps via dockge or portainer) natively within TN to provide the best flexibility when fault finding or developing.

Frigate, via docker, requires the Nvidia container toolkit:

which explains that

Additional configuration is needed for the Docker container to be able to access the NVIDIA GPU. The supported method for this is to install the NVIDIA Container Toolkit and specify the GPU to Docker. How you do this depends on how Docker is being run:

Docker Compose - Nvidia GPU​

services:
  frigate:
    ...
    image: ghcr.io/blakeblackshear/frigate:stable
    deploy:    # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0'] # this is only needed when using multiple GPUs
              count: 1 # number of GPUs
              capabilities: [gpu]

Without the Nvidia container toolkit, there is no safe supported way to implement the docker approach other than developer mode and enabling apt which may be to the detriment of the TrueNAS appliance functionality.

edit - I investigated applications other than just Frigate which could be containerised and which might benefit from the toolkit:

  • DeepStack
  • TensorFlow Serving
  • PyTorch
  • OpenVINO
  • HandBrake
  • Blender
  • Stable Diffusion
2

This is already installed on the TNCE host by default.

nvidia-container-cli list | grep -vFf <(nvidia-container-cli list --libraries)
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
/dev/nvidia-modeset
/dev/nvidia0
/usr/bin/nvidia-smi
/usr/bin/nvidia-debugdump
/usr/bin/nvidia-persistenced
/usr/bin/nvidia-cuda-mps-control
/usr/bin/nvidia-cuda-mps-server
/lib/firmware/nvidia/550.142/gsp_ga10x.bin
/lib/firmware/nvidia/550.142/gsp_tu10x.bin

nvidia-container-cli -V    
cli-version: 1.17.5
lib-version: 1.17.5
build date: 2025-03-07T15:46+00:00
build revision: f23e5e55ea27b3680aef363436d4bcf7659e0bfc
build compiler: x86_64-linux-gnu-gcc-7 7.5.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

You do need to do a custom compose.yaml though to use it like your example above.

3

This is excellent news: thanks very much for showing me this. There are a few Frigate users who will be delighted to learn of this matter.
I tested those two CLI commands and they give me the same information, which proves I ought to be able to use it.

Please can you advise or otherwise point me in the direction of any URLs which show how to configure the yaml - I seem not to be able to get it to work (although knowing my limited skills and knowledge, it could easily be other aspects which are the culprit)?

1 Like
4

What does the official Frigate docs show on configuring the the app with Nvidia? Some apps require additional ENVVARS to be set. You may need to configure the Docker daemon with it’s daemon.json to have the nvidia runtime enabled on the TNCE host. I’ve never done that, so I don’t know what that would look like, but you may be able to setup a PREINIT or POSTINIT job with the nvidia-ctk command below and see if it writes out on next reboot. I know that config doesn’t exist on my 25.04 host without some manipulation… You can also specify the Nvidia runtime in your docker compose as well.

nvidia-ctk runtime configure --runtime=docker

cat /etc/docker/daemon.json
{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

The only other thing is that the Nvidia driver needs to be installed under Apps.

1 Like
5

I ran the command, restarted the docker service and looked at the daemon and I saw a similar result (and showing that the path contains the same runtime):

{
    "data-root": "/mnt/.ix-apps/docker",
    "default-address-pools": [
        {
            "base": "172.17.0.0/12",
            "size": 24
        }
    ],
    "default-runtime": "nvidia",
    "exec-opts": [
        "native.cgroupdriver=cgroupfs"
    ],
    "iptables": true,
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    },
    "storage-driver": "overlay2"
}#

Thanks very much indeed for this information and guidance. There shouldn’t be anything stopping Frigate users from employing a docker approach if they prefer it to the official “app” approach.

1 Like