24.10.1 can't upgrade apps

TrueNAS General
1

after upgrading (fresh installing) 24.10.1, I cannot upgrade my apps. I get one of two errors:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 509, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 556, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1367, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/upgrade.py", line 75, in upgrade
    new_values = self.middleware.call_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1654, in call_sync
    return self.run_coroutine(methodobj(*prepared_call.args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1694, in run_coroutine
    return fut.result()
           ^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/schema_normalization.py", line 34, in normalize_and_validate_values
    dict_obj = await self.middleware.call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1460, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/schema_validation.py", line 44, in validate_values
    await self.validate_question(
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/schema_validation.py", line 67, in validate_question
    await self.validate_question(
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/schema_validation.py", line 87, in validate_question
    await self.middleware.call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1460, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/schema_validation.py", line 141, in validate_port_available_on_node
    if value in await self.middleware.call('app.used_ports') or value in await self.middleware.call(
                                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1460, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ports/ports.py", line 112, in ports_mapping
    for attachment in filter(lambda entry: entry['namespace'] != whitelist_namespace, await self.get_in_use()):
                                                                                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ports/ports.py", line 51, in get_in_use
    used_ports = await delegate.get_ports()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/attachments.py", line 140, in get_ports
    (device['attributes']['bind'], device['attributes']['port']),
     ~~~~~~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'bind'

or

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 509, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 556, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1367, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 55, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/apps/upgrade.py", line 36, in upgrade
    raise CallError('In order to upgrade an app, it must not be in stopped state')
middlewared.service_exception.CallError: [EFAULT] In order to upgrade an app, it must not be in stopped state

The apps are indeed stopped, and i’ve even tried starting them, and then re-stopping them. This is a fresh install of 24.10.1, since for the second time in a row after upgrading the middlewared service does not start. This was working fine prior to 24.10.1 on 24.10.0.2

2

I have the same problem, but just for the nextcloud app.
The following apps just updated fine:
adguard-home
audiobookshelf
collabora
dockge
home-assistant
immich
nginx-proxy-manager
paperless-ngx
plex
searxng
syncthing
tailscale

3

you’re lucky. I dont’ have many, but none of them will update:

plex
tftpd-hpa
smartctl-exporter (prometheuscommunity/smartctl-exporter)
pxewebserver (jitesoft/lighttpd)

4

I just had this same issue… Then I realized that I read the error wrong. Read it again:
“In order to upgrade an app, it must not be in stopped state”

Make sure that the app is running and then try the upgrade. :beers:

5

Tried that, didn’t work. I even tried starting and then stopping again to see if it was stuck. I’m not 100% sure of it, but I believe the cause for me was a full boot disk, due to the audit database ballooning out of control. I couldn’t boot my NAS anymore, it would just die at trying to start middlewared, and hang specifically on the audit plugin. After checking my boot volume, the sqlite3 database for the SUDO audit log was 48GB, consuming all the space. After nuking the DB and enforcing a size limit of like 4GB for the audit config, I am now able to apply updates again.