Cannot change virtual machine pci pass through devices

I’m using a 1660 super GTX card and a P4 gpu acellerator in my T620 Truenas scale server.

Currently, the Ollama app is using the 1660S and I have a Linux mint VM set to use the P4 as a passthrough device for GPU functions etc.

I want to swap over the two cards so the app uses the P4 and the VM uses the 1660S but I’m having trouble getting it done.

The app is set to use Nvidia mode and has gpu “5” listed under the experimental options. I’m assuming that is the id for the 1660S.

The Linux mint VM has the P4 set as a pci pass-through device and has been running fine with no problems.

So,

I tried switching off the Ollama app and adding the 1660S as the pci device for the VM, then removing the P4.

Results? The VM wont startup. It just seems to hang on the VM management screen with the spinning startup wheel.

I removed the PCI devices from the VM and that seemed to let it start, but of course that then had no GPU capability.

I added the P4 back in as the PCI device to the VM exactly as it was before, startup is still hanging.

seemingly if I use either of the cards now as a device, the VM wont start.

Resetting everything back as it was, the ollama app shell under nvidia-smi shows the 1660S as the loaded and active device, so that is working ok, but now the VM is DOA…

Not sure what’s going wrong here. the server is not reporting any issues or logs it seems for the VM and why it’s failing. I’m probably looking in the wrong place but I’ve no clue…

Bar me restarting the whole system to clear any device allocations loaded, I’m drawing a blank…

Any thoughts?

Ta,

K.

sort of got it working (though whilst the 1660S has more power than the P4, it has less memory (6 vs 8GB)). Had to do a few restarts and re-configs of the devices plus whilst the P4 had only one device under the PCI list, the 1660S has 4 that needed adding (GPU, USB, Serial and Hub).

If needs be I can swap back by spooling down this VM (clone) and bringing up the original, and restarting the ollama app.