`[EFAULT] 'vm_name' VM is suspended and can only be resumed/powered off`

Apps and Virtualization
,
1

Arrived in the office this morning to find 9 of my 10 VMs not working. TrueNAS itself was still up. Trying to start any of the VMs resulted in [EFAULT] 'vm_name' VM is suspended and can only be resumed/powered off Anyone ever see this? Know why it might have happened?

I don’t see any UI in TrueNAS to suspend/resume a VM (is there?).

Interestingly the 1 VM that was still running was the one and only one that had the ‘start at boot’ option on. But TrueNAS did not reboot last night, its uptime is days old. Could be a clue.

This is TrueNAS Scale 25.04.2.6.

The full traceback was:


Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 323, in process_method_call
    result = await method.call(app, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 52, in call
    result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 911, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 720, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 93, in wrapped
    result = await func(*args)
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 40, in start
    raise CallError(f'{vm["name"]!r} VM is suspended and can only be resumed/powered off')
middlewared.service_exception.CallError: [EFAULT] 'emu_ctms' VM is suspended and can only be resumed/powered off
2

This has just happened to me again. And now I’m on 25.10.3.1. I’m not certain, but I think it happens when the system is under load. Interestingly, this time only 1 of 10 VMs died, and it’s the 1 that didn’t die last time. Dunno if that’s a coincidence.

3

You’ll have to provide some more info on your system. There’s not enough information here to help diagnose. It seems from the traceback the stuck VMs just went into power save.

4

Check how to use the command ‘virsh list’, so you can check the VM status directly from the command line.

TN used to kill VMs when memory went low (LARC), so I’m really curious to see such code, AND, if the VMs were truly suspended (and can be resumed), as I implemented automatic VM suspend, INSTEAD of shutdown, when rebooting TN, and wish it was an option in TN.

5

It has happened again. This time like the first time: 9/10 VMs are shown as not running.

Regarding memory, the TN web UI shows:

  • Free: 15.8 GiB
  • ZFS Cache: 105.6 GiB
  • Services: 130.2 GiB

I didn’t know about virsh, thanks, I’ve not had time to really read the man page, but I will. I’ve just started work this morning and found your reply and VMs down at the same time. virsh list --all lists nothing at all, not even the 1 VM that is indeed up.

6

These VMs are shown under Virtual Machines and NOT under Containers, right?

List the full virsh command (need to provide specific arguments) used.

7

Correct, these are VMs, not containers.

As I said, the full command I tried was virsh list --all, here’s the full try:

truenas_admin@mynas[~]$ virsh list --all
 Id   Name   State
--------------------

truenas_admin@mynas[~]$ sudo virsh list --all
[sudo] password for truenas_admin:
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
8

By any chance do these suspensions coincide with snapshots for the underlying zvols? Have you installed the qemu-guest-agent in these VMs so they’re aware of snapshots and can quiesce their zvol activity during snapshots? Have you enabled “suspend_on_snapshot” via midclt call vm.update <id> '{ "suspend_on_snapshot": true }' for the ID of each VM?

9

Hmm, that’s an interesting thought. The root dataset has a recursive snapshot task at 03:15 daily. But usually the ping failures are detected before that, like around 01:00 or so.

Interestingly, the 1 VM that often survives is the only one that has an additional snapshot task, which is hourly.

So I think this hypothesis probably doesn’t fit the evidence.

As for the other stuff: no, I don’t have any of that, because I didn’t know about any of it, but it all looks very interesting and I will read about it.

(My Windows VMs do have virtio network drivers installed though, from here. But it’s not only Windows VMs that are getting suspended.)

10

I checked with midclt call vm.query | jq and suspend_on_snapshot is false for all VMs.

I see there is a vm.resume API call. When this bug happens again, I’ll see if I can use that to bring the VM back to life!

11

So this problem reoccurred today, and so I finally tried midclt call vm.status, and indeed the state was ‘suspended’. I then tried midclt call vm.resume and it brought the VM back to life. Yay.

Now I’ve just applied this patch to my system and will see if the problem reoccurs.