Hello Guys,
Recently, i had to setup Windows VM on my NAS and due to rendering needs, i had to use GPU passthrough. Although, the VM works fine, it never works if i restart/shutdown either from the Windows or via TrueNAS>Virtualization section. The system gets halted and i’ve to force reboot the whole NAS using the restart button. I think this is because there is no way to specify the GPU ROM when setting up a GPU passthrough. I can confirm this as i tried the Ubuntu KVM/QEMU on the same machine and when using the GPU ROM, it all works perfect (the restart/shutdown) either from the Windows or via the KVM/QEMU Virtual Machine Manager (GUI).
This impacts the workflow and always restarting the NAS isn’t feasible at all. Let’s suppose you’re working on some files and started the VM. You install an App or a driver that requires restart and when you do that, system halts. It will not come online, until you hard reset the machine (aka force reboot via hardware button).
I manually tried to edit the VM XML and it resulted in success but the changes made to the XML does not persists after reboots.
I’ve used KVM/QEMU on Ubuntu and have tried ESXi, Proxmox, Unraid and they all supported two major things: Specify the GPU ROM and manual editing of XML. Adding these two features in the TrueNAS SCALE will simply help the users to opt more for TrueNAS SCALE rather than opting other virtualization platforms. Imagine you started with TrueNAS which serves your storage need and you now have a need of VM or some apps like tailscale or database in future, so instead of building a new machine or deploying other virtualization for such purpose can be tedious. For such needs, you just have two options: Either build a separate machine for the VM or completely virtualize everything on top of it. Like use ESXi or Proxmox and there you can have more features for the VMs and if you have storage needs, Virtualize TrueNAS as well. I’m sure this can work for some of the users who have such needs, but most of the users are homelabbers or small business owners who are happy with the TrueNAS and they just want to deploy some Apps or VMs to run some services on the same machine, without having the need to setup another one. This can work for one time but when the VM management on TrueNAS is not that flexible like other Virtualization platforms, it just cut off the user experience. I’ve seen several threads where users had to simply switch to Proxmox or ESXi for a better VM management experience. Moreover, managing more machines at home or at workplace (if you are not an administrator), can be hectic sometimes.
If its really not possible to add the functionality (like KVM/QEMU Manager, where you can just go to settings and opt for manual editing of the XML file), due to how TrueNAS is designed, option for adding GPU ROM would be really worth it.
If you would like to have either of these feature in TrueNAS SCALE, please VOTE.
Thanks
You appear to have completely missed the fact that TrueNAS is ditching KVM/QEMU and moving to Incus. Incus is far less flexible and adding some non-standard QEMU options via raw.qemu option is really clunky and prone to just breaking things – Instance options - Incus documentation
I doubt this feature is going to get implemented any time soon, if ever.
1 Like
Dang shit. What did i just read!
This is really sad to hear. Is Incus really better than QEMU/KVM? Either in terms of performance or the flexibility?
Libvirt has more options than Incus, that’s true. Incus is more focused on Virtio devices. But since you can just configure QEMU directly in Incus via raw.qemu and other settings it can do anything QEMU supports if you know how.
It technically uses QEMU/KVM underneath, so it can’t be any better in terms of performance. As for flexibility: no, it’s far more inflexible. It’s meant to make it easy and quick to whip up containers and VMs and thus it takes away a lot of configurability unless you’re willing to mess with raw.qemu
1 Like
Yes, then i don’t see really a need to make a shift to Incus. Rather than moving and which requires far more coding to the existing TrueNAS Code, and some of the Incus feature may require implementation from scratch, wouldn’t it be more better just to improve the existing one when its widely adopted by the users and what would be more good than the fact that users are familiar with QEMU/KVM?
So, is Incus replacing the Docker function on TrueNAS (Apps deployment) or the Virtualization?
Docker functionality is staying, but Incus is replacing the current virtualization stuff. Incus does add the ability to run LXC containers in addition to VMs and Docker containers, so that is a useful feature.
Incus is solution for both LXC containers and QEMU VMs in one.
The out of the box available devices are little more limited in some ways but that’s just because of security and it’s way of doing things.
Either way you can always directly configure qemu which makes it infinitely flexible if you take time to learn it.
It also has its own strengths in comparison to libvirt.
All in all while transition will be difficult it’s worth it and Incus is the better long term solution.
Custom editing won’t happen for several reasons and also what other people already said.
Support for GPU ROM is not entirely out of the question and we will need to gather community interest. Suggest you to rename/repurpose the title/description for this to be considered.
Thank you
1 Like
Hmm. For specifically me, the existing Virtualization and Containers works fine. It would have been more better if the Virtualization was untouched and Docker shifted to Incus. But yeah, i understand.
Hmm. I see.
Yes, like i mentioned, if its not possible to add custom XML editing, just adding support for adding GPU ROM, it would be really nice.
Gotcha. Will do that!
Incus doesn’t do Docker, so that would’ve been impossible.
Docker and Incus are complementary to each other. Both have different purpose.
I think you should first try Incus and check if this problem is present there.
Then it depends if the problem is incus specific or truenas specific.
I don’t know if incus has the ability to set gpu rom. If not you could create issue upstream in Incus Github. If it does and Truenas just doesn’t expose it, then it belongs here.
Oh, yeah. I totally get that.
Umm, how can i use Incus? I’m currently using the latest TrueNAS SCALE and don’t see any option for Incus. Moreover, does Incus has the option to specify the GPU ROM for GPU Passthrough?
Incus it currently only in Fangtooth RC.
But it’s also very early and incomplete implementation.
You can check Incus docs here. Type: gpu - Incus documentation
I don’t know if Incus itself can specify gpu rom. You would need to check it yourself.
Ok looks like I found the base option in QEMU how to add romfile.
Example:
-device virtio-net-pci,netdev=mynet0,bootindex=1,romfile=""
Source
And i dont see it anywhere in Incus docs nor source code.
So I think for this the best solution is requesting in Incus Github adding support for manually setting GPU romfile.
If they add it then Truenas just needs to expose it in UI and middleware.
Is this for any GPU? Or specifically because of the GPUs you are using.
Please confirm which ones are impacted. If other see the same problem with their GPUs, they can add other variants.
I’m using 5700XT. As indicated previously, on the same hardware, the restart and shutdown function either via VM side or the Virtual Machine Manager GUI works fine. But not with TrueNAS and i suspect the issue is due to GPU ROM not being specified.
