Error removing a GPU from isolation

TrueNAS General
,
1

i tried isolating a gpu earlier to pass thru to a VM but that didnt work. Now i just want to be able to use it for apps again and i cannot because it will not allow itself to be removed from isolated gpu devices. this is the error i get:

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 47, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/system_advanced/gpu.py", line 53, in update_gpu_pci_ids
    await self.middleware.call('boot.update_initramfs')
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/boot.py", line 234, in update_initramfs
    raise CallError(f'Failed to update initramfs: {cp.stderr}')
middlewared.service_exception.CallError: [EFAULT] Failed to update initramfs: Failed to update initramfs
Traceback (most recent call last):
  File "/usr/local/bin/truenas-initrd.py", line 252, in <module>
    update_pci_initramfs_config(root),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/truenas-initrd.py", line 188, in update_pci_initramfs_config
    set_readonly(root, False)
  File "/usr/local/bin/truenas-initrd.py", line 46, in set_readonly
    os.rename(os.path.join(root, "usr/local/bin/dpkg"), os.path.join(root, "usr/local/bin/dpkg.bak"))
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/dpkg' -> '/usr/local/bin/dpkg.bak'
2

Hello newguy123,
I am experiencing the same error. My GPU was isolated, and it was passed to a virtual machine without issues. I removed the virtual machine and tried to remove the GPU from isolation. After unchecking the GPU and clicking save the I got an error in line 252 from truenas-initrd.py (update-pci-initramfs_config(root).

image
image776×359 14.8 KB

After refreshing the GPU disappeared from the isolation list and I was able to isolate it again. However, when I try to use (passthrough) it for a new virtual machine I received and error indicating the GPU is not available.

image
image721×255 8.32 KB

I already checked this GPU is not used by any other APP or container.

Were you able to deal with your issue?

3

Not the best solution, but you could try restarting.

Otherwise, it’s probably a bug, and you should perhaps report it if you can repro.

1 Like
4

i ended up reinstalling the OS to fix the error but did not try to reisolate the gpu

5

I restart the system but it did not help. However, the idea of restarting made me to review my bios settings. I enable the ARI support, verify the gpu for passthrough was enable but not as the primary gpu and restart. After this I was able to re-isolate the gpu and added it from the list of devices (editing the virtual machine devices).

After this, I am able to passthrough the gpu into the virtual machine.

I tested to see if I could replicate the passtrough disabling/enabling gpu for the VM.
Now It works fine.

So, my recommendation is to verify in the bios the gpu to be used as the passthrough device is not used by the main system. Probably is not required for all systems but it may necesary in some cases.
Also, in the bios be sure all settings for virtualization are enable.

Second, if you are, for example, moving the passtrough gpu from one VM to another VM in truenas, delete the device (vga and sound devices) from the first VM, then remove the gpu from the isolated devices.

You may receive a warning message and the gpu will be seem to persist in isolation but if you refresh the instance you will notice the gpu is actually not isolated.

Finally, to passthrough the gpu follow the regular steps. Isolate the gpu first and enable the gpu passthrough in the VM or add the devices if the VM already exist (display and audio if applicable)
I hope this helps to avoid a reinstalation of the OS.

2 Likes