Sounds a bit like same problem I have: Cannot assign GPU to App - nvidia-device-plugin CrashLoopBackOff
I’m hoping 24.04.1 later this week will fix this, but as it is not accepted as a problem på ix i stil think it will not.
If you create a support ticket, ix might at least her convinced its a problem and start finding a solution.