A replication task that I had is suddenly missing and as soon as I create a new task, I get the following error
Error log
Error Name: EINVAL
Error Code: 22
Reason: 3 validation errors for ReplicationQueryResult
result.list[ReplicationQueryResultItem].0.periodic_snapshot_tasks.0.state
String should have at most 1024 characters [type=string_too_long, input_value='{"state": {"state": "ERR...pshots were created."}}', input_type=str]
For further information visit https://errors.pydantic.dev/2.9/v/string_too_long
result.ReplicationQueryResultItem
Input should be a valid dictionary or instance of ReplicationQueryResultItem [type=model_type, input_value=[{'id': 5, 'target_datase...d_dataset_keys': False}], input_type=list]
For further information visit https://errors.pydantic.dev/2.9/v/model_type
result.int
Input should be a valid integer [type=int_type, input_value=[{'id': 5, 'target_datase...d_dataset_keys': False}], input_type=list]
For further information visit https://errors.pydantic.dev/2.9/v/int_type
Error Class: ValidationError
Trace: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 360, in process_method_call
result = await method.call(app, id_, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 69, in call
return await self._dump_result(app, methodobj, result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 72, in _dump_result
return self.middleware.dump_result(self.serviceobj, methodobj, app, result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 862, in dump_result
return serialize_result(new_style_returns_model, result, expose_secrets)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/handler/result.py", line 13, in serialize_result
return model(result=result).model_dump(
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/pydantic/main.py", line 212, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ReplicationQueryResult
result.list[ReplicationQueryResultItem].0.periodic_snapshot_tasks.0.state
String should have at most 1024 characters [type=string_too_long, input_value='{"state": {"state": "ERR...pshots were created."}}', input_type=str]
For further information visit https://errors.pydantic.dev/2.9/v/string_too_long
result.ReplicationQueryResultItem
Input should be a valid dictionary or instance of ReplicationQueryResultItem [type=model_type, input_value=[{'id': 5, 'target_datase...d_dataset_keys': False}], input_type=list]
For further information visit https://errors.pydantic.dev/2.9/v/model_type
result.int
Input should be a valid integer [type=int_type, input_value=[{'id': 5, 'target_datase...d_dataset_keys': False}], input_type=list]
For further information visit https://errors.pydantic.dev/2.9/v/int_type
I’m also unable to delete the periodic task linked to that replication task as that too throws an error:
Error message
Error Name: EFAULT
Error Code: 14
Reason: [EFAULT] You are deleting the last periodic snapshot task bound to enabled replication task 'tank/data,tank/home,tank/shares - replication' which will break it. Please, disable that replication task first.
Error Class: CallError
Trace: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 360, in process_method_call
result = await method.call(app, id_, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 57, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 954, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 771, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 196, in delete
return await self.middleware._call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 771, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 214, in nf
rv = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 108, in wrapped
result = await func(*args)
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/snapshot.py", line 276, in do_delete
raise CallError(
middlewared.service_exception.CallError: [EFAULT] You are deleting the last periodic snapshot task bound to enabled replication task 'tank/data,tank/home,tank/shares - replication' which will break it. Please, disable that replication task first.
How do I recover the task or delete it by force in such a case?