I am using TrueNAS ElectricEel-24.10.2. I am getting error when i am trying to replicate data between two TrueNAS nodes. The replications job rubs for some time but stops with giving below mentioned error
Connection closed by 10.X.X.X
Diagnosing these issues requires good documentation of how you you have set up the system and replication task. Its only then that someone might be able to spot then config issue…
I’d recomend setting up a dummy dataset according the docs… see if you can set that up or document what you have done. User Permissions can be important.
The exact issue is given below
[2025/05/13 15:00:00] INFO [Thread-1374] [zettarepl.paramiko.replication_task__task_4] Connected (version 2.0, client OpenSSH_9.2p1)
[2025/05/13 15:00:00] INFO [Thread-1374] [zettarepl.paramiko.replication_task__task_4] Authentication (publickey) successful!
[2025/05/13 15:00:00] INFO [replication_task__task_4] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/05/13 15:00:01] INFO [replication_task__task_4] [zettarepl.replication.run] For replication task ‘task_4’: doing push from ‘Pool1/cloudstack1’ to ‘Pool1/backup’ of snapshot=‘auto-2025-05-08_11-27’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/05/13 15:02:01] WARNING [replication_task__task_4] [zettarepl.replication.run] For task ‘task_4’ at attempt 1 recoverable replication error RecoverableReplicationError(‘Connection closed by 10.50.50.81 port 22\nBroken pipe.’)
[2025/05/13 15:02:01] INFO [replication_task__task_4] [zettarepl.replication.run] After recoverable error sleeping for 1 seconds
[2025/05/13 15:02:02] INFO [replication_task__task_4] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/05/13 15:02:02] INFO [replication_task__task_4] [zettarepl.replication.run] For replication task ‘task_4’: doing push from ‘Pool1/cloudstack1’ to ‘Pool1/backup’ of snapshot=‘auto-2025-05-08_11-27’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/05/13 15:04:02] WARNING [replication_task__task_4] [zettarepl.replication.run] For task ‘task_4’ at attempt 2 recoverable replication error RecoverableReplicationError(‘Connection closed by 10.50.50.81 port 22\nBroken pipe.’)
[2025/05/13 15:04:02] INFO [replication_task__task_4] [zettarepl.replication.run] After recoverable error sleeping for 2 seconds
… 2 more lines …
[2025/05/13 15:06:05] WARNING [replication_task__task_4] [zettarepl.replication.run] For task ‘task_4’ at attempt 3 recoverable replication error RecoverableReplicationError(‘Connection closed by 10.50.50.81 port 22\nBroken pipe.’)
[2025/05/13 15:06:05] INFO [replication_task__task_4] [zettarepl.replication.run] After recoverable error sleeping for 4 seconds
[2025/05/13 15:06:09] INFO [replication_task__task_4] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/05/13 15:06:10] INFO [replication_task__task_4] [zettarepl.replication.run] For replication task ‘task_4’: doing push from ‘Pool1/cloudstack1’ to ‘Pool1/backup’ of snapshot=‘auto-2025-05-08_11-27’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/05/13 15:08:10] WARNING [replication_task__task_4] [zettarepl.replication.run] For task ‘task_4’ at attempt 4 recoverable replication error RecoverableReplicationError(‘Connection closed by 10.50.50.81 port 22\nBroken pipe.’)
[2025/05/13 15:08:10] INFO [replication_task__task_4] [zettarepl.replication.run] After recoverable error sleeping for 8 seconds
[2025/05/13 15:08:18] INFO [replication_task__task_4] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/05/13 15:08:18] INFO [replication_task__task_4] [zettarepl.replication.run] For replication task ‘task_4’: doing push from ‘Pool1/cloudstack1’ to ‘Pool1/backup’ of snapshot=‘auto-2025-05-08_11-27’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/05/13 15:10:18] WARNING [replication_task__task_4] [zettarepl.replication.run] For task ‘task_4’ at attempt 5 recoverable replication error RecoverableReplicationError(‘Connection closed by 10.50.50.81 port 22\nBroken pipe.’)
[2025/05/13 15:10:18] ERROR [replication_task__task_4] [zettarepl.replication.run] Failed replication task ‘task_4’ after 5 retries
You’ll need to provide more setup information… its failing to recover from an error, but there is no clear reason for the error.
I have two identical dell servers OEMR XL R740xd servers with TrueNAS ElectricEel-24.10.2 installed on both of those servers. I am starting the replication task between both servers, primary server IP 10.50.50.82 & secondary server IP 10.50.50.81. The replication task runs from primary server to secondary server using ssh. The task starts & endsup giving the failed error.
The task fro primary server uses root user to execute replication task. Let me know if I misses anything.
Thanks & Regards
Amandeep Singh
Can anyone help on this issue ?