Hello everybody, i have a big mistery between my hands, i am totally unable to perform a replication over a router level vpn(wireguard). So here is the setup
1-Both servers are running truenas scale 25.04.2.6
2-Both servers seems to be accesible from each other and the conection stable enough:
traceroute to 192.168.2.100 (192.168.2.100), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 0.129 ms 0.090 ms 0.088 ms
2 10.25.35.1 (10.25.35.1) 26.466 ms 26.441 ms 26.659 ms
3 192.168.2.100 (192.168.2.100) 30.767 ms 30.743 ms 30.719 ms
traceroute to 192.168.2.100 (192.168.2.100), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 0.129 ms 0.090 ms 0.088 ms
2 10.25.35.1 (10.25.35.1) 26.466 ms 26.441 ms 26.659 ms
3 192.168.2.100 (192.168.2.100) 30.767 ms 30.743 ms 30.719 ms
Connecting to host 192.168.1.178, port 5201
[ 5] local 192.168.2.100 port 34020 connected to 192.168.1.178 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 12.5 MBytes 105 Mbits/sec 0 1.12 MBytes
[ 5] 1.00-2.00 sec 10.0 MBytes 83.9 Mbits/sec 0 1.12 MBytes
[ 5] 2.00-3.00 sec 11.2 MBytes 94.4 Mbits/sec 0 1.12 MBytes
[ 5] 3.00-4.00 sec 11.2 MBytes 94.4 Mbits/sec 0 1.12 MBytes
[ 5] 4.00-5.00 sec 10.0 MBytes 83.9 Mbits/sec 0 1.12 MBytes
[ 5] 5.00-6.00 sec 11.2 MBytes 94.4 Mbits/sec 0 1.12 MBytes
[ 5] 6.00-7.00 sec 10.0 MBytes 83.9 Mbits/sec 0 1.12 MBytes
[ 5] 7.00-8.00 sec 11.2 MBytes 94.4 Mbits/sec 0 1.12 MBytes
[ 5] 8.00-9.00 sec 11.2 MBytes 94.4 Mbits/sec 0 1.12 MBytes
[ 5] 9.00-10.00 sec 10.0 MBytes 83.9 Mbits/sec 0 1.12 MBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 109 MBytes 91.2 Mbits/sec 0 sender
[ 5] 0.00-10.04 sec 106 MBytes 88.9 Mbits/sec receiveriperf Done.
3-The VPN is a wireguard conection between an ubiquiti router and a opensense one and should be transparent for both servers in teory at least.
The issue:
Both trying to pull or push , it works fine for around 30 mins, then the comunication stops(you can see it on the network stadistics for example) till it ramdonly timeouts. Small datasets/snapshots can be transfered corercly on this time, but big(hundreds of GB) snapshots will never finish. This is the first and main issue that i am trying to solve
The second issue seems to be a truenas bug, since the listener keeps open on the other side(in the case of a push) and the dataset locked, so the process needs to be manually kill or the server reset to be able to retry. After that the process will resume from the begining of the snapshot that was being transfer, and since is a couble of hunders of gb is never going to end(it will always timeout before finish) so is an infinite loop. Is basically this the same as on last Lawrence Systems video called “How to Fix TrueNAS ZFS Replication Failure by Clearing The Resume Token” but 100% of the times
i woudl post the /var/log/zettarepl.log but i think is too clouded with all my atemps ¬¬, but for a push over ssh with not root user ends up with:
zettarepl.log exctract of error 1
[2025/12/03 10:55:07] WARNING [replication_task__task_3] [zettarepl.replication.run] For task ‘task_3’ at attempt 1 recoverable replication error RecoverableReplicationError(‘Timeout, server 192.168.2.100 not responding.\nBroken pipe.’)
[2025/12/03 10:55:07] INFO [replication_task__task_3] [zettarepl.replication.run] After recoverable error sleeping for 1 seconds
[2025/12/03 10:55:09] INFO [replication_task__task_3] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/12/03 10:55:09] INFO [replication_task__task_3] [zettarepl.replication.run] Resuming replication for destination dataset ‘RemoteDatos/NotRootUserName/HDD 2’
[2025/12/03 10:55:09] INFO [replication_task__task_3] [zettarepl.replication.run] For replication task ‘task_3’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=None incremental_base=None include_intermediate=None receive_resume_token=‘1-111ca1c3a5-120-789c636064000310a501c49c50360710a715e5e7a69766a6304001ef87326319d55f960a40363b92bafca4acd4e4120606a7f5107518f26969c5a9250c7000926743924faa2c492d06b10fa68a62d35f920f718595ee8d77abf598777920c97382e5f312735319185c124bf28b3df3d213738a13f53d5c5c149c1373325312531c124b4bf2758d0c8c4c750d0d8174bc8181ae8101c41e6e0684ff93f3730b8a528b8bf3b3116e050071982c3e’ encryption=False
[2025/12/03 10:55:10] WARNING [replication_task__task_3] [zettarepl.replication.partially_complete_state] Specified receive_resume_token, but received an error: contains partially-complete state. Allowing ZFS to catch up
[2025/12/03 10:56:10] INFO [replication_task__task_3] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
Extrar of a try with netcat
2025/12/11 14:53:19] INFO [MainThread] [zettarepl.scheduler.clock] Interrupted
[2025/12/11 14:53:19] INFO [MainThread] [zettarepl.zettarepl] Scheduled tasks: [<Replication Task ‘task_7’>]
[2025/12/11 14:53:20] INFO [Thread-485] [zettarepl.paramiko.replication_task__task_7] Connected (version 2.0, client OpenSSH_9.2p1)
[2025/12/11 14:53:21] INFO [Thread-485] [zettarepl.paramiko.replication_task__task_7] Authentication (publickey) successful!
[2025/12/11 14:53:21] INFO [Thread-487] [zettarepl.paramiko.retention] Connected (version 2.0, client OpenSSH_9.2p1)
[2025/12/11 14:53:21] INFO [Thread-487] [zettarepl.paramiko.retention] Authentication (publickey) successful!
[2025/12/11 14:53:21] ERROR [retention] [zettarepl.replication.task.snapshot_owner] Failed to list snapshots with <Shell(<SSH Transport(ingalsabackup@192.168.2.100)>)>: DatasetDoesNotExistException(1, “cannot open ‘RemoteDatos/NotRootUserName/HDD 1’: dataset does not exist\n”). Assuming remote has no snapshots
[2025/12/11 14:53:21] INFO [retention] [zettarepl.zettarepl] Retention destroying local snapshots:
[2025/12/11 14:53:22] INFO [replication_task__task_7] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/12/11 14:53:22] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-11-28_00-00’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:24] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-11-29_00-00’ incremental_base=‘auto-2025-11-28_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:25] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-11-30_00-00’ incremental_base=‘auto-2025-11-29_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:26] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-01_00-00’ incremental_base=‘auto-2025-11-30_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:27] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-02_00-00’ incremental_base=‘auto-2025-12-01_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:29] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-03_00-00’ incremental_base=‘auto-2025-12-02_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:30] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-04_00-00’ incremental_base=‘auto-2025-12-03_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:31] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-05_00-00’ incremental_base=‘auto-2025-12-04_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:32] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-06_00-00’ incremental_base=‘auto-2025-12-05_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:34] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-07_00-00’ incremental_base=‘auto-2025-12-06_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:35] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-08_00-00’ incremental_base=‘auto-2025-12-07_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:36] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-09_00-00’ incremental_base=‘auto-2025-12-08_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:37] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-10_00-00’ incremental_base=‘auto-2025-12-09_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:38] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 1’ to ‘RemoteDatos/NotRootUserName/HDD 1’ of snapshot=‘auto-2025-12-11_00-00’ incremental_base=‘auto-2025-12-10_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 14:53:41] INFO [replication_task__task_7] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/12/11 14:53:41] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-11-28_00-00’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:04] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-11-29_00-00’ incremental_base=‘auto-2025-11-28_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:06] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-11-30_00-00’ incremental_base=‘auto-2025-11-29_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:08] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-01_00-00’ incremental_base=‘auto-2025-11-30_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:11] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-02_00-00’ incremental_base=‘auto-2025-12-01_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:13] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-03_00-00’ incremental_base=‘auto-2025-12-02_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:15] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-04_00-00’ incremental_base=‘auto-2025-12-03_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:16] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-05_00-00’ incremental_base=‘auto-2025-12-04_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:17] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-06_00-00’ incremental_base=‘auto-2025-12-05_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:19] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-07_00-00’ incremental_base=‘auto-2025-12-06_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:21] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-08_00-00’ incremental_base=‘auto-2025-12-07_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:25] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-09_00-00’ incremental_base=‘auto-2025-12-08_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:26] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-10_00-00’ incremental_base=‘auto-2025-12-09_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:30] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 2’ to ‘RemoteDatos/NotRootUserName/HDD 2’ of snapshot=‘auto-2025-12-11_00-00’ incremental_base=‘auto-2025-12-10_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:33] INFO [replication_task__task_7] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/12/11 15:19:33] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-11-28_00-00’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:35] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-11-29_00-00’ incremental_base=‘auto-2025-11-28_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:36] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-11-30_00-00’ incremental_base=‘auto-2025-11-29_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:37] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-01_00-00’ incremental_base=‘auto-2025-11-30_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:38] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-02_00-00’ incremental_base=‘auto-2025-12-01_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:40] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-03_00-00’ incremental_base=‘auto-2025-12-02_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:41] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-04_00-00’ incremental_base=‘auto-2025-12-03_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:42] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-05_00-00’ incremental_base=‘auto-2025-12-04_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:43] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-06_00-00’ incremental_base=‘auto-2025-12-05_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:44] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-07_00-00’ incremental_base=‘auto-2025-12-06_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:46] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-08_00-00’ incremental_base=‘auto-2025-12-07_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:47] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-09_00-00’ incremental_base=‘auto-2025-12-08_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:48] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-10_00-00’ incremental_base=‘auto-2025-12-09_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:49] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 3’ to ‘RemoteDatos/NotRootUserName/HDD 3’ of snapshot=‘auto-2025-12-11_00-00’ incremental_base=‘auto-2025-12-10_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:51] INFO [replication_task__task_7] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/12/11 15:19:52] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-11-28_00-00’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:54] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-11-29_00-00’ incremental_base=‘auto-2025-11-28_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:55] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-11-30_00-00’ incremental_base=‘auto-2025-11-29_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:57] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-01_00-00’ incremental_base=‘auto-2025-11-30_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:58] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-02_00-00’ incremental_base=‘auto-2025-12-01_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:19:59] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-03_00-00’ incremental_base=‘auto-2025-12-02_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:00] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-04_00-00’ incremental_base=‘auto-2025-12-03_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:01] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-05_00-00’ incremental_base=‘auto-2025-12-04_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:03] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-06_00-00’ incremental_base=‘auto-2025-12-05_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:04] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-07_00-00’ incremental_base=‘auto-2025-12-06_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:05] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-08_00-00’ incremental_base=‘auto-2025-12-07_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:06] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-09_00-00’ incremental_base=‘auto-2025-12-08_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:07] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-10_00-00’ incremental_base=‘auto-2025-12-09_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:09] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 4’ to ‘RemoteDatos/NotRootUserName/HDD 4’ of snapshot=‘auto-2025-12-11_00-00’ incremental_base=‘auto-2025-12-10_00-00’ include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:20:11] INFO [replication_task__task_7] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/12/11 15:20:11] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 5’ to ‘RemoteDatos/NotRootUserName/HDD 5’ of snapshot=‘auto-2025-11-28_00-00’ incremental_base=None include_intermediate=False receive_resume_token=None encryption=False
[2025/12/11 15:26:52] ERROR [replication_task__task_7.dataset_size_observer] [zettarepl.replication.dataset_size_observer] Dataset size observer error: SSHException(‘Timeout opening channel.’)
[2025/12/11 15:27:32] ERROR [replication_task__task_7.dataset_size_observer] [zettarepl.replication.dataset_size_observer] Dataset size observer error: SSHException(‘Timeout opening channel.’)
[2025/12/11 15:27:45] WARNING [Thread-485] [zettarepl.paramiko.replication_task__task_7] Success for unrequested channel! [??]
[2025/12/11 15:27:45] WARNING [Thread-485] [zettarepl.paramiko.replication_task__task_7] Success for unrequested channel! [??]
[2025/12/11 15:34:17] WARNING [replication_task__task_7] [zettarepl.replication.run] For task ‘task_7’ at attempt 1 recoverable replication error RecoverableReplicationError(‘Timeout, server 192.168.2.100 not responding.\nBroken pipe.’)
[2025/12/11 15:34:17] INFO [replication_task__task_7] [zettarepl.replication.run] After recoverable error sleeping for 1 seconds
[2025/12/11 15:34:19] INFO [replication_task__task_7] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/12/11 15:34:19] INFO [replication_task__task_7] [zettarepl.replication.run] Resuming replication for destination dataset ‘RemoteDatos/NotRootUserName/HDD 5’
[2025/12/11 15:34:19] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 5’ to ‘RemoteDatos/NotRootUserName/HDD 5’ of snapshot=None incremental_base=None include_intermediate=None receive_resume_token=‘1-f5b05c223-f8-789c636064000310a500c4ec50360710e72765a526973030c841d560c8a7a515a7968064e6c1e5d990e4932a4b528b81f489d865a6d8f497e4a79766a63030b47b4577b5f9eb267b20c97382e5f312735319185c124bf28b3df3d213738a13f53d5c5c14dc538b52f39233131d124b4bf2758d0c8c4c750d0d758d2ce20d0c740d0cc0f6703320fc959c9f5b50945a5c9c9fcd000700e85b25f2’ encryption=False
[2025/12/11 15:34:20] WARNING [replication_task__task_7] [zettarepl.replication.partially_complete_state] Specified receive_resume_token, but received an error: contains partially-complete state. Allowing ZFS to catch up
[2025/12/11 15:35:20] INFO [replication_task__task_7] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
[2025/12/11 15:35:20] INFO [replication_task__task_7] [zettarepl.replication.run] Resuming replication for destination dataset ‘RemoteDatos/NotRootUserName/HDD 5’
[2025/12/11 15:35:20] INFO [replication_task__task_7] [zettarepl.replication.run] For replication task ‘task_7’: doing push from ‘Datos/HDD 5’ to ‘RemoteDatos/NotRootUserName/HDD 5’ of snapshot=None incremental_base=None include_intermediate=None receive_resume_token=‘1-11294fe7c0-f8-789c636064000310a500c4ec50360710e72765a526973030c841d560c8a7a515a796806436c0e5d990e4932a4b528b817447f87a536cfa4bf2d34b33531818dabda2bbdafc75933d90e439c1f27989b9a90c0c2e8925f9c59e79e98939c589fa1e2e2e0aeea945a979c999890e89a525f9ba460646a6ba8686ba4616f10606ba0606607bb81910fe4acecf2d284a2d2ececf66800300d18f25c7’ encryption=False
[2025/12/11 16:04:42] WARNING [replication_task__task_7] [zettarepl.replication.run] For task ‘task_7’ at attempt 2 recoverable replication error RecoverableReplicationError(‘resume token contents:\nnvlist version: 0\n\tobject = 0x11e\n\toffset = 0xb00000\n\tbytes = 0x35af5788\n\ttoguid = 0x632d4f868a5b4a87\n\ttoname = Datos/HDD 5@auto-2025-11-28_00-00\n\tcompressok = 1\nTimeout, server 192.168.2.100 not responding.\nBroken pipe.’)
[2025/12/11 16:04:42] INFO [replication_task__task_7] [zettarepl.replication.run] After recoverable error sleeping for 2 seconds
[2025/12/11 16:04:45] INFO [replication_task__task_7] [zettarepl.replication.pre_retention] Pre-retention destroying snapshots:
What i tried:
First i tried adjusting the mtu on both ends to 1400 or even lower but no results. Then tried to increase the keepalive on the ssh connection. Both with no results.
Tried both pull or push, tried with or without encription(netcat), always a timeout ramdomly stop after 5-40min of the start of the transfer.
Tried both with root user, and with normal user with sudo permisions(doubt is the issue since some small snapshots/datasets are correctly transfer only big one fails)
Does anyone have more ideas to try? i am totally out of them 