`[EFAULT] 'vm_name' VM is suspended and can only be resumed/powered off`

Arrived in the office this morning to find 9 of my 10 VMs not working. TrueNAS itself was still up. Trying to start any of the VMs resulted in [EFAULT] 'vm_name' VM is suspended and can only be resumed/powered off Anyone ever see this? Know why it might have happened?

I don’t see any UI in TrueNAS to suspend/resume a VM (is there?).

Interestingly the 1 VM that was still running was the one and only one that had the ‘start at boot’ option on. But TrueNAS did not reboot last night, its uptime is days old. Could be a clue.

This is TrueNAS Scale 25.04.2.6.

The full traceback was:


Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 323, in process_method_call
    result = await method.call(app, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 52, in call
    result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 911, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 720, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 93, in wrapped
    result = await func(*args)
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 40, in start
    raise CallError(f'{vm["name"]!r} VM is suspended and can only be resumed/powered off')
middlewared.service_exception.CallError: [EFAULT] 'emu_ctms' VM is suspended and can only be resumed/powered off