The container is the large language AI model (LLM) from Meta. Deploying in a container provides isolation from the internet, allowing local and private research and development. It’s on Docker hub here:
I’ve looked to see if the the resource is being seen from Portainer, a container app, and the Apps GUI provided by TRueNAS. So far it’s not obvious to me, or perhaps I’m missing something.
But from a TrueNAS OS user perspective, precisely the way I see my system has 12 CPU cores, RAM and an LAN connection, and additionally provided the work load percentage and temperature of each core, RAM % utilized, and LAN speed, that same information provided for newly installed hardware by OS monitoring is an intermediate to confirming the OS has GPU cores for hire. I’ve looked for this GPU info from the OS, but always have to back to the Nvidia tools.
I will dig into how the container can detect new resources and let you know.
The TrueNAS system denied this request. Is there anyway to get around this, as these tools are, as you say, likely not part of the base install for TrueNAS container support. This is an exploding use case for containers and should be supported. And I thought I saw somebody got Ollama working on TrueNAS, with GPU support.
Wondering if video transcoding and this toolkit could be supported? I really wanted to run this LLM app on a bare metal container, but suppose I could run a VM. But then don’t I have the same GPU pass-thru issue, where the toolkit is required for this application?
Hope I’m not stuck, as this was one of the main reasons I’ve been patiently waiting for TrueNAS to support containers VIA Docker.
apt is disabled by default because truenas is an appliance os and you can break stuff by updating and installing the underlying linux packages.
You can enable developer mode which lets you use apt at your own risk.
Edit: You could setup a nspawn-container (scale-jail) and install the nvidia-toolkit and driver there and run the app in docker.
I have yet to migrate to the native docker option because it lacks some features i’d really like to have, which are offered by the jail (option for separate ip per app, free port 53 for blocky or pihole just to name some)
As I said, apt is not available - users are not allowed to just install additional software on the base OS.
Reason for this change was that users broke their entire TrueNAS system by manually running updates with apt. Thats why the devs disabled it alltogether.
There appears to be a way to get it back, but that will flag the entire system as unsupported which is visible in the debug logs when you file a bug or ask for support.
TrueNAS is not the right choice for virtual machines / containers with special requirements like additional apt packages.
You also have to ask yourself if you even want your NAS to take on such specific tasks on top of its main task (being a NAS).
If you are limited to a single machine, I’d install proxmox for your VMs and then also run TrueNAS as a VM on that Proxmox host to which you pass through an HBA.
That way you get nice seperation of your services while still using just one physical system.
/edit:
thinking about it, you might have success deploying a linux VM on TrueNAS to which you pass through the GPU(s), and then install ollama inside that VM.
I do have just the one machine and over built it because I wanted to support containers and VMs (dumpster dive find: Dell PowerEdge R410, from MIT, upgraded to 12 Xeon cores, 16TB of HD space, 256GB of RAM and an Nvidia Tesla P4 GPU card). It’s the nicest PC system I’ve built to date, and it’s a fabulous NAS, running TrueNAS Scale! It’s amazing what folks throw into the trash these days!
But your point about TrueNAS Scale making a great NAS appliance, I completely missed. My bad! TrueNAS Scale is a great NAS! I just got excited and ahead of myself wanting more, what with the VM and container support. I didn’t realize access to the Linux OS was limited in the way you describe. Guess I got greedy, given I used to run an SQL Server container on an old Mac MINI for development work that stopped working when I upgraded Docker Desktop. Yikes!
So I was in need of a replacement. The trash to treasure, about $800 dollars and TrueNAS Scale was hands down way better than a Mac replacement and way, way, way more affordable!
But you’ve got the wheels spinning and I’m reconsidering my options now – may attempt another dumpster dive or try out that Linux VM idea you floated.
Thanks again for the discussion. Much appreciated.
I’m trying to get Ollama working in a container too, with no success. What I’m getting back from this thread is it’s not possible because apt has been disable by the TrueNAS dev team, and this is needed in order to load the required Nvidia drivers and toolkit for a more performant Ollama experience.
My question is are you running Ollama at higher performance than an only CPU solution can provide, given you got it to work in a container or VM, with GPU pass-thru? If so, how was this achieved?
The Problem is not that you can’t intsall nvidia drivers, that should be done either through the gui or an api call.
The api call should be midctl call -job docker.update ‘{“nvidia”: true}’
The Problem most likely is that the nvidia-toolkit is not included in the truenas installation. You can create a feature request and if enough people vote for it it will get included.
I didn’t understand your post re: create a jail, and install the Nvidia toolkit there. I’m very new to TrueNAS, and it’s been decades since I’ve had to do anything Unix related, but will look into it.
As for submitting a ticket, I think that’s a great idea! Sign me up! Folks wanting to use their TrueNAS appliance in new and exciting ways, and leveraging their existing hardware or new hardware is progress – a goal the entire community willingly spends time and effort to pursue.
I’ll see if I can find the workaround. But both ComfyUI and Ollama run utilizing a rtx a4000 w/out any issues. I run everything via docker compose, and no issues.
…I’ll look for the workaround this weekend for you.
Is there any known hacky workaround to do this without reinstalling the entire app again? Would really rather not have to do that for my Immich/Plex/other apps that require my GPU. Although since everything is now Docker it shouldn’t be too hard to get the data back
