Upgrade error KeyError: 'fk_network_lagginterface_lagg_interface_id_network_interfaces'

Hi there,

i get the following error output on upgrade TrueNAS Core from TrueNAS-24.04.2.3 → TrueNAS-24.10.0.2:

I have not found anything regarding this and not really sure what is going on.
I already checked the freenas-v1.db sqlite config database but nothing looks wrong with that to me.
(for context: I needed to do some manual adjustments in the database back some years ago but I think it was only deleting unneeded config entries due to the database growing endlessly)

Anybody have an idea where I can start investigating?

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 511, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update.py", line 286, in update
    await self.middleware.call('update.install', job, os.path.join(location, 'update.sqsh'), options)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1428, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install_linux.py", line 32, in install
    self.middleware.call_sync("update.install_scale", mounted, progress_callback, options)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1600, in call_sync
    return methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install.py", line 52, in install_scale
    self._execute_truenas_install(mounted, command, progress_callback)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install.py", line 83, in _execute_truenas_install
    raise CallError(result or f"Abnormal installer process termination with code {p.returncode}")
middlewared.service_exception.CallError: [EFAULT] Command ['chroot', '/tmp/tmp6z83m448', 'migrate'] failed with exit code 1: + DATABASE=/data/freenas-v1.db
+ python3 -c import os; import sys; sys.path.remove("") if "" in sys.path else None; import middlewared; print(os.path.dirname(middlewared.__spec__.origin))
+ cd /usr/lib/python3/dist-packages/middlewared
+ FREENAS_DATABASE=/data/freenas-v1.db alembic upgrade head
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/psutil/_common.py", line 399, in wrapper
    return cache[key]
           ~~~~~^^^^^
KeyError: (('/proc',), frozenset())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 285, in <module>
    set_scputimes_ntuple("/proc")
  File "/usr/lib/python3/dist-packages/psutil/_common.py", line 401, in wrapper
    ret = cache[key] = fun(*args, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 268, in set_scputimes_ntuple
    with open_binary('%s/stat' % procfs_path) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/psutil/_common.py", line 728, in open_binary
    return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/proc/stat'
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 6a7c2281f48e -> 968d515e63e7, merge migration
INFO  [alembic.runtime.migration] Running upgrade 968d515e63e7 -> 7836261b2f64, Add enable learning flag
INFO  [alembic.runtime.migration] Running upgrade 7836261b2f64 -> 1a6fc6735dc2, Cloud backup keep last
INFO  [alembic.runtime.migration] Running upgrade 1a6fc6735dc2 -> 14974a858948, Add global system ID
INFO  [alembic.runtime.migration] Running upgrade 14974a858948, ea024b5dff95 -> d7243bc7bab9, Merge
INFO  [alembic.runtime.migration] Running upgrade d7243bc7bab9, 423e4c21c28d -> 3cb1ccef4ade, empty message
INFO  [alembic.runtime.migration] Running upgrade 3cb1ccef4ade, d774066c6c0c -> 4f11cc19bb9c, empty message
INFO  [alembic.runtime.migration] Running upgrade 4f11cc19bb9c, f38c2bbe776a -> 135a7e02cbec, Merge migration for changes in SMB-related fields from 24.04
INFO  [alembic.runtime.migration] Running upgrade 135a7e02cbec -> 0dc9c3f51393, Remove swap configuration from system advanced
INFO  [alembic.runtime.migration] Running upgrade 0dc9c3f51393 -> 91724c382023, Apps catalog integration
INFO  [alembic.runtime.migration] Running upgrade 91724c382023 -> d8bfbf4e277e, add smb encryption parameter
INFO  [alembic.runtime.migration] Running upgrade d8bfbf4e277e -> 1307a8e6a8b6, Add login_banner column
INFO  [alembic.runtime.migration] Running upgrade 1307a8e6a8b6 -> 81b8bae8fb11, Remove FTP rootlogin
INFO  [alembic.runtime.migration] Running upgrade 81b8bae8fb11 -> 4b0b7ba46e63, Add flag to see if image update is requried
INFO  [alembic.runtime.migration] Running upgrade 4b0b7ba46e63 -> 5654da8713d1, NVIDIA
INFO  [alembic.runtime.migration] Running upgrade 5654da8713d1 -> 7b13df980355, Cascade delete LAGG interface configuration when network interface configuration is deleted
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/alembic/operations/batch.py", line 687, in drop_constraint
    const = self.named_constraints.pop(const.name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'fk_network_lagginterface_lagg_interface_id_network_interfaces'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/alembic", line 33, in <module>
    sys.exit(load_entry_point('alembic==1.8.1.dev0', 'console_scripts', 'alembic')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/alembic/config.py", line 590, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/usr/lib/python3/dist-packages/alembic/config.py", line 584, in main
    self.run_cmd(cfg, options)
  File "/usr/lib/python3/dist-packages/alembic/config.py", line 561, in run_cmd
    fn(
  File "/usr/lib/python3/dist-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/usr/lib/python3/dist-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/lib/python3/dist-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/middlewared/alembic/env.py", line 167, in <module>
    run_migrations_online()
  File "/usr/lib/python3/dist-packages/middlewared/alembic/env.py", line 161, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/lib/python3/dist-packages/alembic/runtime/environment.py", line 853, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python3/dist-packages/alembic/runtime/migration.py", line 623, in run_migrations
    step.migration_fn(**kw)
  File "/usr/lib/python3/dist-packages/middlewared/alembic/versions/24.10/2024-08-16_10-01_lagg_on_delete_cascade.py", line 21, in upgrade
    with op.batch_alter_table('network_lagginterface', schema=None) as batch_op:
  File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/middlewared/alembic/env.py", line 45, in batch_alter_table_impl
    with original_batch_alter_table(self, *args, **kwargs) as result:
  File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/alembic/operations/base.py", line 381, in batch_alter_table
    impl.flush()
  File "/usr/lib/python3/dist-packages/alembic/operations/batch.py", line 157, in flush
    fn(*arg, **kw)
  File "/usr/lib/python3/dist-packages/alembic/operations/batch.py", line 695, in drop_constraint
    raise ValueError("No such constraint: '%s'" % const.name)
ValueError: No such constraint: 'fk_network_lagginterface_lagg_interface_id_network_interfaces'

Best,
Felix

Hi,

I’ve got the same error when I try to update TrueNAS from TrueNAS-24.04.2.5 to TrueNAS-24.10.1. I tried it many times, but always the same error. I already tried to make a fresh 24.04 install and restore the config. Without any success.

Can somebody explain the problem?

Did you manage to resolve the issue or did you reconfigure your system?

It would be useful to document your hardware and report a bug. We can see if the config file is corrupted or there is something else going on?