The biggest Problem with gpu passthrough was that docker expected a different mount path for the gpu.
Docker expects the mount at:
/proc/driver/nvidia/gpus/0000:01:00.0 #where 0000:01:00.0 is the gpu id
but the passthrough checkbox mounts the gpu at
/dev/nvidia0
I found an article online which suggests a symlink from /dev/nvidia0 to /proc/driver/nvidia/gpus/0000:01:00.0
But it needs to be executed every time before the docker service starts. This can be achieved by a new systemd entry.
Create a new unit file at /etc/systemd/system/fix-gpu-passthrough.service
with the following content:
[Unit]
Description=Creates Symlink required for LXC/Nvidia to Docker passthrough
Before=docker.service
[Service]
User=root
Group=root
ExecStart=/bin/bash -c 'mkdir -p /proc/driver/nvidia/gpus && ln -s /dev/nvidia0 /proc/driver/nvidia/gpus/0000:01:00.0' #replace with your own gpu id
Type=oneshot
[Install]
WantedBy=multi-user.target
save and restart docker
sudo systemctl daemon-reload
sudo systemctl enable fix-gpu-passthrough
sudo systemctl restart docker
Without the symlink i was getting hit by this error message:
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as ‘legacy’
nvidia-container-cli: mount error: stat failed: /proc/driver/nvidia/gpus/0000:01:00.0: no such file or directory: unknown
The rest was just permissions, because i forgot to add the truenas_container_unpriv_root user to all Datasets i passed through.