Electric Eel Broke Virtualization?

Apps and Virtualization
1

After upgrading to 24.10.0, my VM’s aren’t working… I did a bit of googling and found a few posts relating to permissions errors, or removing and re adding a virtual disk, but nothing seems to solve. Curiously I also can not create a new VM, as I get a similar error.

This is what I get when trying to start my most important vm called “Ubuntu”

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 45, in _check_add_domain
    self._add(self._vm_from_name(vm_name)['id'])
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 16, in _add
    self._add_with_vm_data(vm)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 19, in _add_with_vm_data
    self.vms[vm['name']] = VMSupervisor(vm, self.middleware)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 30, in __init__
    self.update_domain()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 66, in update_domain
    self.__define_domain()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 99, in __define_domain
    if not self.LIBVIRT_CONNECTION.defineXML(vm_xml):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/libvirt.py", line 4495, in defineXML
    raise libvirtError('virDomainDefineXML() failed')
libvirt.libvirtError: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/bin/qemu-system-x86_64 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/qmp-HSDIW2/qmp.monitor,server=on,wait=off -pidfile /var/lib/libvirt/qemu/qmp-HSDIW2/qmp.pid -daemonize) unexpected fatal signal 7

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 208, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1526, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1457, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 58, in start
    await self.middleware.run_in_thread(self._start, vm['name'])
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1361, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 67, in _start
    self._check_add_domain(vm_name)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 47, in _check_add_domain
    raise CallError(f'Unable to define domain for {vm_name}: {e}')
middlewared.service_exception.CallError: [EFAULT] Unable to define domain for Ubuntu: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/bin/qemu-system-x86_64 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/qmp-HSDIW2/qmp.monitor,server=on,wait=off -pidfile /var/lib/libvirt/qemu/qmp-HSDIW2/qmp.pid -daemonize) unexpected fatal signal 7

This is what I get when I try to create a new VM called purple

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 208, in call_method
    result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1526, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1457, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 230, in create
    return await self.middleware._call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1457, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 261, in nf
    rv = await func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vms.py", line 213, in do_create
    await self.middleware.run_in_thread(self._add, vm_id)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1361, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 16, in _add
    self._add_with_vm_data(vm)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 19, in _add_with_vm_data
    self.vms[vm['name']] = VMSupervisor(vm, self.middleware)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 30, in __init__
    self.update_domain()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 66, in update_domain
    self.__define_domain()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 99, in __define_domain
    if not self.LIBVIRT_CONNECTION.defineXML(vm_xml):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/libvirt.py", line 4495, in defineXML
    raise libvirtError('virDomainDefineXML() failed')
libvirt.libvirtError: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/bin/qemu-system-x86_64 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/qmp-2FVIW2/qmp.monitor,server=on,wait=off -pidfile /var/lib/libvirt/qemu/qmp-2FVIW2/qmp.pid -daemonize) unexpected fatal signal 7
1 Like
2

After lots more research, poking, prodding, slamming my head against my keyboard, I could not find a resolution. I took a config backup, swapped my boot drive, clean installed electric eel from a usb (had already updated zfs flags so no going back to a previous version), restored config, and viola! Everything works immediately.

Long story short, the problem was over my head so I took the nuclear option.

3

I’m having a similar problem, but don’t have your option of a “nuclear reinstall.” It’s definitely a bug introduced when upgrading from 24.10-RC2 to 24.10.0, though. And since the package management is “off limits” since it isn’t exposed via middleware, I’m not even sure that reinstalling the relevant packages would help.

4

I do wish I understood the inner workings of not only TrueNAS, but Linux in general so could have spent more time trying to figure this out. It looks like 24.10 caused a lot more problems than anticipated.

5

We don’t have any current bug tickets related to this error. If you would like to submit a bug report with a debug file we’d be happy to investigate: Issue Reporting in Jira | TrueNAS Documentation Hub

The last time we did have a bug report with this error message ([EFAULT] Unable to define domain) was nearly a year ago and the cause turned out to be a detached usb device configured on the VM: [NAS-124992] - Jira

6

Whatever the problem actually was, and it seems to have been related to the Python code, it went away when upgrading to 24.10.1 and 24.10.2. The bug submission process is non-trivial for non-enterprise customers, and I actually gave up on it after about 2 hours of trying to submit a bug report. (shrug)

I think the broader issue is that there’s not a lot of easily-accessible information about what the middleware does or what it controls. As an unrelated example, I have a PR I may submit (if the process isn’t excessively burdensome) to fix a number of issues with some underlying assumptions the middleware makes about the chronyd daemon and the actual locations of the RTC dump files in AppArmor that routinely get clobbered even when the modified files are specifically designated as local overrides.

Not to sidetrack the issue, but I consider the VM problem solved, but I think this type of thread is symptomatic of undocumented middleware assumptions, and there’s no easy way to look some of them up or modify them without some very weird tweaks inside the UI. I know the middleware is basically iX system’s “secret sauce” for differentiating from plain ol’ ZFS, but unless I missed something (which I admit is certainly possible) I think this is mostly a documentation and package management issue regarding 24.10.0 specifically.

7

The bug submission process is really quite straightforward if you’re doing it through TrueNAS, which is ordinarily the best way to do it anyway. Just click on the “Send feedback” button–it’s in the upper-right corner, just to the right of the iXSystems logo:

image
image830×108 9.16 KB

In the window that pops up, next to “I would like to:”, click on “Report a bug.”

image
image1084×1314 60.9 KB

Fill in the subject and message, leave “Attach debug” checked, and you’ll be able to log into (and create an account if necessary at) Jira, their bug tracker.

4 Likes