To use a GPU inside a VM, need to:
- Isolate it under ˋsystem settings → advanced → Isolated GPU Device(s)ˋ
- Reboot
- Create The VM (if you are already having issues with the system complaining don’t use the “easy” select option in the edit section)
- Open the devices menu of your VM and pass through both the dedicated GPU and it’s integrated audio controller (if they are listed separately, add two devices)
- Start up the VM
Windows 11 will not work out of the Box, because it needs a virtual TPM. That can be added via the command line and there is an open Feature Request you can vote on Add "Machine Type Q35" and "TPM 2.0" options to VMs - #17 by SeaFox
But for now Windows 10 is better for making sure everything else works
Edit: Issues beyond that might have to do with IOMMU groups or your GPU having even more “devices” bundled together that all have to be passed through together. It’s also worth checking whether IOMMU, SR-IOV, and other virtualization stuff is explicitly enabled in the UEFI. Especially IOMMU when set to auto often does not behave well and should be explicitly enabled