I am running TrueNAS 25.04.2.6 and trying to upgrade to 25.10.1 and am getting a barrage of errors as follows hereupon. Can anyone suggest what steps I might need to take to remediate my server such that it can be upgraded?
[EFAULT] Error: Command [‘chroot’, ‘/tmp/tmpv4kdi2zw’, ‘migrate’] failed with exit code 1: + DATABASE=/data/freenas-v1.db + PWENC_SECRET=/data/pwenc_secret + python3 -c import os; import sys; sys.path.remove(“”) if “” in sys.path else None; import middlewared; print(os.path.dirname(middlewared._spec_.origin)) + cd /usr/lib/python3/dist-packages/middlewared + FREENAS_DATABASE=/data/freenas-v1.db FREENAS_PWENC_SECRET=/data/pwenc_secret alembic upgrade head INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade 9ada77affbb9 → 5fda0931889d, Drop vm tables INFO [alembic.runtime.migration] Running upgrade 5fda0931889d → ed9a781ec0c6, Remove disk_smartoptions INFO [alembic.runtime.migration] Running upgrade ed9a781ec0c6, 0257529fa6d5 → 616c19f82016, Merge migration for adding uid/gid 568 idmap INFO [alembic.runtime.migration] Running upgrade 616c19f82016 → a156968d5cbb, Remove syslog_tls_certificate_authority INFO [alembic.runtime.migration] Running upgrade a156968d5cbb → 940b79ac591f, API key revoked reason INFO [alembic.runtime.migration] Running upgrade 940b79ac591f → 801eb4df44ce, Rename uq_truenas_enclosurelabel_encid to uq_enclosure_label_encid INFO [alembic.runtime.migration] Running upgrade 801eb4df44ce, df0bffcf1595 → a34e4c124c25, Merge migration for incus storage pools INFO [alembic.runtime.migration] Running upgrade a34e4c124c25 → cf1f98f4c3b1, Remove system.general language column. INFO [alembic.runtime.migration] Running upgrade cf1f98f4c3b1 → 9a5b103ec2e4, Remove CA plugin INFO [alembic.runtime.migration] Running upgrade 9a5b103ec2e4 → f15312414057, Remove SMART support INFO [alembic.runtime.migration] Running upgrade f15312414057, 249b95f63f76 → d7e3a916db65, Merge migration after adding account policy columns INFO [alembic.runtime.migration] Running upgrade d7e3a916db65 → 595b38b52541, Disable password when empty string is used. INFO [alembic.runtime.migration] Running upgrade 595b38b52541 → cafccdf50053, Cloud backup cache path INFO [alembic.runtime.migration] Running upgrade cafccdf50053 → 08febd74cdf9, Add nvmet INFO [alembic.runtime.migration] Running upgrade 08febd74cdf9 → aeae837fb2d5, Periodic snapshot task state INFO [alembic.runtime.migration] Running upgrade aeae837fb2d5, 30c9619bf9e7 → c227e49be4d8, Merge INFO [alembic.runtime.migration] Running upgrade c227e49be4d8 → dae46dda9606, NVMe target does not require trsvcid for FC INFO [alembic.runtime.migration] Running upgrade dae46dda9606 → f8b0ab7c2275, Migrate directory services to new schema Traceback (most recent call last): File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1900, in _execute_context self.dialect.do_execute( File “/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py”, line 736, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: no such index: ix_directoryservice_ldap_ldap_certificate_id The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/usr/bin/alembic”, line 33, in sys.exit(load_entry_point(‘alembic==1.8.1.dev0’, ‘console_scripts’, ‘alembic’)()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/alembic/config.py”, line 590, in main CommandLine(prog=prog).main(argv=argv) File “/usr/lib/python3/dist-packages/alembic/config.py”, line 584, in main self.run_cmd(cfg, options) File “/usr/lib/python3/dist-packages/alembic/config.py”, line 561, in run_cmd fn( File “/usr/lib/python3/dist-packages/alembic/command.py”, line 322, in upgrade script.run_env() File “/usr/lib/python3/dist-packages/alembic/script/base.py”, line 569, in run_env util.load_python_file(self.dir, “env.py”) File “/usr/lib/python3/dist-packages/alembic/util/pyfiles.py”, line 94, in load_python_file module = load_module_py(module_id, path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/alembic/util/pyfiles.py”, line 110, in load_module_py spec.loader.exec_module(module) # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “”, line 940, in exec_module File “”, line 241, in _call_with_frames_removed File “/usr/lib/python3/dist-packages/middlewared/alembic/env.py”, line 167, in run_migrations_online() File “/usr/lib/python3/dist-packages/middlewared/alembic/env.py”, line 161, in run_migrations_online context.run_migrations() File “”, line 8, in run_migrations File “/usr/lib/python3/dist-packages/alembic/runtime/environment.py”, line 853, in run_migrations self.get_context().run_migrations(**kw) File “/usr/lib/python3/dist-packages/alembic/runtime/migration.py”, line 623, in run_migrations step.migration_fn(**kw) File “/usr/lib/python3/dist-packages/middlewared/alembic/versions/25.10/2025-05-17_12-57_migrate-ds.py”, line 434, in upgrade with op.batch_alter_table(‘directoryservice_ldap’, schema=None) as batch_op: File “/usr/lib/python3.11/contextlib.py”, line 144, in _exit_ next(self.gen) File “/usr/lib/python3/dist-packages/middlewared/alembic/env.py”, line 45, in batch_alter_table_impl with original_batch_alter_table(self, *args, **kwargs) as result: File “/usr/lib/python3.11/contextlib.py”, line 144, in _exit_ next(self.gen) File “/usr/lib/python3/dist-packages/alembic/operations/base.py”, line 381, in batch_alter_table impl.flush() File “/usr/lib/python3/dist-packages/alembic/operations/batch.py”, line 111, in flush fn(*arg, **kw) File “/usr/lib/python3/dist-packages/alembic/ddl/impl.py”, line 395, in drop_index self._exec(schema.DropIndex(index)) File “/usr/lib/python3/dist-packages/alembic/ddl/impl.py”, line 195, in _exec return conn.execute(construct, multiparams) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1380, in execute return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/sqlalchemy/sql/ddl.py”, line 80, in _execute_on_connection return connection._execute_ddl( ^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1472, in _execute_ddl ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1943, in _execute_context self._handle_dbapi_exception( File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 2124, in _handle_dbapi_exception util.raise_( File “/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py”, line 211, in raise_ raise exception File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1900, in _execute_context self.dialect.do_execute( File “/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py”, line 736, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such index: ix_directoryservice_ldap_ldap_certificate_id [SQL: DROP INDEX ix_directoryservice_ldap_ldap_certificate_id] (Background on this error at: https://sqlalche.me/e/14/e3q8)
upon clicking “More info” I saw this:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 515, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 560, in __run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 174, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/update.py”, line 287, in update
await self.middleware.call(‘update.install’, job, os.path.join(location, ‘update.sqsh’), options)
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1005, in call
return await self.call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 731, in call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/update/install_linux.py", line 50, in install
self.middleware.call_sync(“update.install_scale”, mounted, progress_callback, options)
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1041, in call_sync
return methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/update/install.py”, line 61, in install_scale
self.execute_truenas_install(mounted, command, progress_callback)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update/install.py", line 92, in _execute_truenas_install
raise CallError(result or f"Abnormal installer process termination with code {p.returncode}")
middlewared.service_exception.CallError: [EFAULT] Error: Command [‘chroot’, ‘/tmp/tmpv4kdi2zw’, ‘migrate’] failed with exit code 1: + DATABASE=/data/freenas-v1.db
- PWENC_SECRET=/data/pwenc_secret
- python3 -c import os; import sys; sys.path.remove(“”) if “” in sys.path else None; import middlewared; print(os.path.dirname(middlewared.spec.origin))
- cd /usr/lib/python3/dist-packages/middlewared
- FREENAS_DATABASE=/data/freenas-v1.db FREENAS_PWENC_SECRET=/data/pwenc_secret alembic upgrade head
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 9ada77affbb9 → 5fda0931889d, Drop vm tables
INFO [alembic.runtime.migration] Running upgrade 5fda0931889d → ed9a781ec0c6, Remove disk_smartoptions
INFO [alembic.runtime.migration] Running upgrade ed9a781ec0c6, 0257529fa6d5 → 616c19f82016, Merge migration for adding uid/gid 568 idmap
INFO [alembic.runtime.migration] Running upgrade 616c19f82016 → a156968d5cbb, Remove syslog_tls_certificate_authority
INFO [alembic.runtime.migration] Running upgrade a156968d5cbb → 940b79ac591f, API key revoked reason
INFO [alembic.runtime.migration] Running upgrade 940b79ac591f → 801eb4df44ce, Rename uq_truenas_enclosurelabel_encid to uq_enclosure_label_encid
INFO [alembic.runtime.migration] Running upgrade 801eb4df44ce, df0bffcf1595 → a34e4c124c25, Merge migration for incus storage pools
INFO [alembic.runtime.migration] Running upgrade a34e4c124c25 → cf1f98f4c3b1, Remove system.general language column.
INFO [alembic.runtime.migration] Running upgrade cf1f98f4c3b1 → 9a5b103ec2e4, Remove CA plugin
INFO [alembic.runtime.migration] Running upgrade 9a5b103ec2e4 → f15312414057, Remove SMART support
INFO [alembic.runtime.migration] Running upgrade f15312414057, 249b95f63f76 → d7e3a916db65, Merge migration after adding account policy columns
INFO [alembic.runtime.migration] Running upgrade d7e3a916db65 → 595b38b52541, Disable password when empty string is used.
INFO [alembic.runtime.migration] Running upgrade 595b38b52541 → cafccdf50053, Cloud backup cache path
INFO [alembic.runtime.migration] Running upgrade cafccdf50053 → 08febd74cdf9, Add nvmet
INFO [alembic.runtime.migration] Running upgrade 08febd74cdf9 → aeae837fb2d5, Periodic snapshot task state
INFO [alembic.runtime.migration] Running upgrade aeae837fb2d5, 30c9619bf9e7 → c227e49be4d8, Merge
INFO [alembic.runtime.migration] Running upgrade c227e49be4d8 → dae46dda9606, NVMe target does not require trsvcid for FC
INFO [alembic.runtime.migration] Running upgrade dae46dda9606 → f8b0ab7c2275, Migrate directory services to new schema
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1900, in _execute_context
self.dialect.do_execute(
File “/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py”, line 736, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such index: ix_directoryservice_ldap_ldap_certificate_id
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/bin/alembic”, line 33, in
sys.exit(load_entry_point(‘alembic==1.8.1.dev0’, ‘console_scripts’, ‘alembic’)())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/alembic/config.py”, line 590, in main
CommandLine(prog=prog).main(argv=argv)
File “/usr/lib/python3/dist-packages/alembic/config.py”, line 584, in main
self.run_cmd(cfg, options)
File “/usr/lib/python3/dist-packages/alembic/config.py”, line 561, in run_cmd
fn(
File “/usr/lib/python3/dist-packages/alembic/command.py”, line 322, in upgrade
script.run_env()
File “/usr/lib/python3/dist-packages/alembic/script/base.py”, line 569, in run_env
util.load_python_file(self.dir, “env.py”)
File “/usr/lib/python3/dist-packages/alembic/util/pyfiles.py”, line 94, in load_python_file
module = load_module_py(module_id, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/alembic/util/pyfiles.py”, line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “”, line 940, in exec_module
File “”, line 241, in _call_with_frames_removed
File “/usr/lib/python3/dist-packages/middlewared/alembic/env.py”, line 167, in
run_migrations_online()
File “/usr/lib/python3/dist-packages/middlewared/alembic/env.py”, line 161, in run_migrations_online
context.run_migrations()
File “”, line 8, in run_migrations
File “/usr/lib/python3/dist-packages/alembic/runtime/environment.py”, line 853, in run_migrations
self.get_context().run_migrations(**kw)
File “/usr/lib/python3/dist-packages/alembic/runtime/migration.py”, line 623, in run_migrations
step.migration_fn(**kw)
File “/usr/lib/python3/dist-packages/middlewared/alembic/versions/25.10/2025-05-17_12-57_migrate-ds.py”, line 434, in upgrade
with op.batch_alter_table(‘directoryservice_ldap’, schema=None) as batch_op:
File “/usr/lib/python3.11/contextlib.py”, line 144, in exit
next(self.gen)
File “/usr/lib/python3/dist-packages/middlewared/alembic/env.py”, line 45, in batch_alter_table_impl
with original_batch_alter_table(self, *args, **kwargs) as result:
File “/usr/lib/python3.11/contextlib.py”, line 144, in exit
next(self.gen)
File “/usr/lib/python3/dist-packages/alembic/operations/base.py”, line 381, in batch_alter_table
impl.flush()
File “/usr/lib/python3/dist-packages/alembic/operations/batch.py”, line 111, in flush
fn(*arg, **kw)
File “/usr/lib/python3/dist-packages/alembic/ddl/impl.py”, line 395, in drop_index
self._exec(schema.DropIndex(index))
File “/usr/lib/python3/dist-packages/alembic/ddl/impl.py”, line 195, in _exec
return conn.execute(construct, multiparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1380, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/sqlalchemy/sql/ddl.py”, line 80, in _execute_on_connection
return connection._execute_ddl(
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1472, in _execute_ddl
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1943, in _execute_context
self.handle_dbapi_exception(
File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 2124, in handle_dbapi_exception
util.raise(
File “/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py”, line 211, in raise
raise exception
File “/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py”, line 1900, in _execute_context
self.dialect.do_execute(
File “/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py”, line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such index: ix_directoryservice_ldap_ldap_certificate_id
[SQL:
DROP INDEX ix_directoryservice_ldap_ldap_certificate_id]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
One additional note is that LDAP seems to no longer be able to connect to my FreeIPA servers (wherein it used to be able to do so). However, I am interested to troubleshoot the upgrade first prior to anything else.
Thanks to anyone that can help.