Hi, I’m using truenas scale as a homelaber.
For offsite backup I use the truenas zfs replication with quite low throughput (similar to this thread) of 2,x MiB/s, which I would like to increase.
The two machines are connected via tailscale (because CGNAT on both ends). The source WAN connection has 500Mbit/s upload, the target has 100Mbit/s nominal download speed, speed tests usually measure around 80Mbit/s. Replication load on the network is only around 22Mbit/s. Tailscale ping is 30ms, CPU and drives are mostly idle on both systems during replication, no core shows exceptional load. Memory is still >16GB free.
What, to my amateur eyes, seems to be the culprit is that iperf3 measures quickly degrading network speed when using only one stream, ending up at around 20-30Mbit/s after a little while. As soon as the number of streams is increased to 4, 80Mbit/s are maintained.
Since I have no influence over the ISPs network equipment, is there anything I can do to increase speed? How do I exclude that the problem lies with one of the routers? I’ve had a router of the same making as the one on the target system (Fritzbox 7530AX) fail after it was no longer able to switch more than 50Mbit/s for some weeks. However, the percentage of repeated packages during replication is below .1% and I do not have physical access to the target system or its network.
ChatGPT said that zfs replication does not support multiple streams. Is that so? I heard that truenas 26.04 uses a newer zfs version than 25.10 (which I am using) which is supposed to increase replication speed. Could a system upgrade to 26.04 benefit my situation as well?
Thank you!
TrueNAS 26 is currently on Beta 3. You can switch the Update train in the GUI and update if you are willing to take the risk on using it before a full release. If you update your pool features, you will not be able to roll back to an earlier version. I don’t know if there are updated pool features between 25 & 26 but it is worth warning about.
tailscale + zfs replication capping around 20mbit while iperf looks fine usually means the send side is chatting a lot for small records. for me keeping compression off when the dataset is already compressed and testing one large dataset alone helps more than jumping to a beta train.
Not sure iperf is fine. One stream is only good for a couple of seconds. For more reliable speed it needs to run in parallel.
One Stream
iperf3 -c 100.126.xxx.xxx -R -t 20
Connecting to host , port 5201
Reverse mode, remote host is sending
[ 5] local port 37584 connected to port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 7.46 MBytes 62.6 Mbits/sec
[ 5] 1.00-2.00 sec 5.27 MBytes 44.2 Mbits/sec
[ 5] 2.00-3.00 sec 3.45 MBytes 29.0 Mbits/sec
[ 5] 3.00-4.00 sec 2.60 MBytes 21.8 Mbits/sec
[ 5] 4.00-5.00 sec 1.89 MBytes 15.9 Mbits/sec
[ 5] 5.00-6.00 sec 1.98 MBytes 16.6 Mbits/sec
[ 5] 6.00-7.00 sec 1.53 MBytes 12.8 Mbits/sec
[ 5] 7.00-8.00 sec 1.46 MBytes 12.3 Mbits/sec
[ 5] 8.00-9.00 sec 2.03 MBytes 17.0 Mbits/sec
[ 5] 9.00-10.00 sec 1.93 MBytes 16.2 Mbits/sec
[ 5] 10.00-11.00 sec 2.28 MBytes 19.1 Mbits/sec
[ 5] 11.00-12.00 sec 1.75 MBytes 14.7 Mbits/sec
[ 5] 12.00-13.00 sec 2.05 MBytes 17.2 Mbits/sec
[ 5] 13.00-14.00 sec 1.48 MBytes 12.4 Mbits/sec
[ 5] 14.00-15.00 sec 1.48 MBytes 12.4 Mbits/sec
[ 5] 15.00-16.00 sec 1.99 MBytes 16.7 Mbits/sec
[ 5] 16.00-17.00 sec 2.01 MBytes 16.8 Mbits/sec
[ 5] 17.00-18.00 sec 1.39 MBytes 11.7 Mbits/sec
[ 5] 18.00-19.00 sec 1.24 MBytes 10.4 Mbits/sec
[ 5] 19.00-20.00 sec 1.77 MBytes 14.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.03 sec 48.8 MBytes 20.4 Mbits/sec 37 sender
[ 5] 0.00-20.00 sec 47.0 MBytes 19.7 Mbits/sec receiver
Six streams in parallel
iperf3 -c 100.126.xxx.xxx -R -P 6 -t 20
Connecting to host , port 5201
Reverse mode, remote host is sending
[ 5] local port 32850 connected to port 5201
[ 7] local port 32852 connected to port 5201
[ 9] local port 32854 connected to port 5201
[ 11] local port 32862 connected to port 5201
[ 13] local port 32876 connected to port 5201
[ 15] local port 32892 connected to port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.82 MBytes 15.2 Mbits/sec
[ 7] 0.00-1.00 sec 2.34 MBytes 19.6 Mbits/sec
[ 9] 0.00-1.00 sec 1.31 MBytes 11.0 Mbits/sec
[ 11] 0.00-1.00 sec 1.33 MBytes 11.2 Mbits/sec
[ 13] 0.00-1.00 sec 965 KBytes 7.91 Mbits/sec
[ 15] 0.00-1.00 sec 1.31 MBytes 11.0 Mbits/sec
[SUM] 0.00-1.00 sec 9.05 MBytes 75.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 1.41 MBytes 11.9 Mbits/sec
[ 7] 1.00-2.00 sec 2.67 MBytes 22.4 Mbits/sec
[ 9] 1.00-2.00 sec 1.51 MBytes 12.7 Mbits/sec
[ 11] 1.00-2.00 sec 1.27 MBytes 10.7 Mbits/sec
[ 13] 1.00-2.00 sec 985 KBytes 8.07 Mbits/sec
[ 15] 1.00-2.00 sec 1.46 MBytes 12.3 Mbits/sec
[SUM] 1.00-2.00 sec 9.30 MBytes 78.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 1.40 MBytes 11.7 Mbits/sec
[ 7] 2.00-3.00 sec 2.40 MBytes 20.1 Mbits/sec
[ 9] 2.00-3.00 sec 1.39 MBytes 11.7 Mbits/sec
[ 11] 2.00-3.00 sec 1.26 MBytes 10.5 Mbits/sec
[ 13] 2.00-3.00 sec 1.06 MBytes 8.88 Mbits/sec
[ 15] 2.00-3.00 sec 1.39 MBytes 11.7 Mbits/sec
[SUM] 2.00-3.00 sec 8.90 MBytes 74.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 1.34 MBytes 11.2 Mbits/sec
[ 7] 3.00-4.00 sec 2.52 MBytes 21.1 Mbits/sec
[ 9] 3.00-4.00 sec 1.47 MBytes 12.3 Mbits/sec
[ 11] 3.00-4.00 sec 910 KBytes 7.46 Mbits/sec
[ 13] 3.00-4.00 sec 1.08 MBytes 9.08 Mbits/sec
[ 15] 3.00-4.00 sec 1.57 MBytes 13.2 Mbits/sec
[SUM] 3.00-4.00 sec 8.86 MBytes 74.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 1.41 MBytes 11.8 Mbits/sec
[ 7] 4.00-5.00 sec 2.75 MBytes 23.1 Mbits/sec
[ 9] 4.00-5.00 sec 993 KBytes 8.13 Mbits/sec
[ 11] 4.00-5.00 sec 1.11 MBytes 9.27 Mbits/sec
[ 13] 4.00-5.00 sec 1.01 MBytes 8.48 Mbits/sec
[ 15] 4.00-5.00 sec 1.86 MBytes 15.6 Mbits/sec
[SUM] 4.00-5.00 sec 9.11 MBytes 76.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 7] 5.00-6.00 sec 2.46 MBytes 20.6 Mbits/sec
[ 9] 5.00-6.00 sec 1.11 MBytes 9.35 Mbits/sec
[ 11] 5.00-6.00 sec 1.37 MBytes 11.5 Mbits/sec
[ 13] 5.00-6.00 sec 932 KBytes 7.63 Mbits/sec
[ 15] 5.00-6.00 sec 1.63 MBytes 13.6 Mbits/sec
[SUM] 5.00-6.00 sec 8.73 MBytes 73.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 1.26 MBytes 10.5 Mbits/sec
[ 7] 6.00-7.00 sec 2.09 MBytes 17.5 Mbits/sec
[ 9] 6.00-7.00 sec 1.17 MBytes 9.80 Mbits/sec
[ 11] 6.00-7.00 sec 1.36 MBytes 11.4 Mbits/sec
[ 13] 6.00-7.00 sec 1.31 MBytes 11.0 Mbits/sec
[ 15] 6.00-7.00 sec 1.54 MBytes 13.0 Mbits/sec
[SUM] 6.00-7.00 sec 8.73 MBytes 73.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 1.06 MBytes 8.85 Mbits/sec
[ 7] 7.00-8.00 sec 2.04 MBytes 17.1 Mbits/sec
[ 9] 7.00-8.00 sec 1.14 MBytes 9.53 Mbits/sec
[ 11] 7.00-8.00 sec 1.40 MBytes 11.8 Mbits/sec
[ 13] 7.00-8.00 sec 1.85 MBytes 15.5 Mbits/sec
[ 15] 7.00-8.00 sec 1.26 MBytes 10.6 Mbits/sec
[SUM] 7.00-8.00 sec 8.74 MBytes 73.3 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 1.26 MBytes 10.6 Mbits/sec
[ 7] 8.00-9.00 sec 1.67 MBytes 14.0 Mbits/sec
[ 9] 8.00-9.00 sec 1.03 MBytes 8.65 Mbits/sec
[ 11] 8.00-9.00 sec 1.31 MBytes 11.0 Mbits/sec
[ 13] 8.00-9.00 sec 2.10 MBytes 17.6 Mbits/sec
[ 15] 8.00-9.00 sec 979 KBytes 8.02 Mbits/sec
[SUM] 8.00-9.00 sec 8.33 MBytes 69.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 1.26 MBytes 10.6 Mbits/sec
[ 7] 9.00-10.00 sec 1.77 MBytes 14.8 Mbits/sec
[ 9] 9.00-10.00 sec 1.27 MBytes 10.7 Mbits/sec
[ 11] 9.00-10.00 sec 1.38 MBytes 11.6 Mbits/sec
[ 13] 9.00-10.00 sec 2.29 MBytes 19.2 Mbits/sec
[ 15] 9.00-10.00 sec 1.08 MBytes 9.09 Mbits/sec
[SUM] 9.00-10.00 sec 9.06 MBytes 76.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 10.00-11.00 sec 1.34 MBytes 11.2 Mbits/sec
[ 7] 10.00-11.00 sec 1.62 MBytes 13.6 Mbits/sec
[ 9] 10.00-11.00 sec 1.58 MBytes 13.3 Mbits/sec
[ 11] 10.00-11.00 sec 1.34 MBytes 11.3 Mbits/sec
[ 13] 10.00-11.00 sec 2.49 MBytes 20.9 Mbits/sec
[ 15] 10.00-11.00 sec 819 KBytes 6.71 Mbits/sec
[SUM] 10.00-11.00 sec 9.18 MBytes 77.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 11.00-12.00 sec 1.36 MBytes 11.4 Mbits/sec
[ 7] 11.00-12.00 sec 1.71 MBytes 14.3 Mbits/sec
[ 9] 11.00-12.00 sec 1.42 MBytes 11.9 Mbits/sec
[ 11] 11.00-12.00 sec 1.06 MBytes 8.87 Mbits/sec
[ 13] 11.00-12.00 sec 1.97 MBytes 16.6 Mbits/sec
[ 15] 11.00-12.00 sec 1.06 MBytes 8.92 Mbits/sec
[SUM] 11.00-12.00 sec 8.58 MBytes 72.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 12.00-13.00 sec 1.01 MBytes 8.47 Mbits/sec
[ 7] 12.00-13.00 sec 1.53 MBytes 12.8 Mbits/sec
[ 9] 12.00-13.00 sec 1.79 MBytes 15.0 Mbits/sec
[ 11] 12.00-13.00 sec 1.46 MBytes 12.3 Mbits/sec
[ 13] 12.00-13.00 sec 1.41 MBytes 11.8 Mbits/sec
[ 15] 12.00-13.00 sec 1.18 MBytes 9.93 Mbits/sec
[SUM] 12.00-13.00 sec 8.38 MBytes 70.3 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 13.00-14.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 7] 13.00-14.00 sec 1.18 MBytes 9.87 Mbits/sec
[ 9] 13.00-14.00 sec 2.06 MBytes 17.3 Mbits/sec
[ 11] 13.00-14.00 sec 1.78 MBytes 15.0 Mbits/sec
[ 13] 13.00-14.00 sec 1.27 MBytes 10.6 Mbits/sec
[ 15] 13.00-14.00 sec 1.01 MBytes 8.51 Mbits/sec
[SUM] 13.00-14.00 sec 8.31 MBytes 69.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 14.00-15.00 sec 1.05 MBytes 8.82 Mbits/sec
[ 7] 14.00-15.00 sec 1.32 MBytes 11.1 Mbits/sec
[ 9] 14.00-15.00 sec 1.92 MBytes 16.1 Mbits/sec
[ 11] 14.00-15.00 sec 1.28 MBytes 10.8 Mbits/sec
[ 13] 14.00-15.00 sec 1.60 MBytes 13.4 Mbits/sec
[ 15] 14.00-15.00 sec 1.36 MBytes 11.4 Mbits/sec
[SUM] 14.00-15.00 sec 8.53 MBytes 71.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 15.00-16.00 sec 895 KBytes 7.33 Mbits/sec
[ 7] 15.00-16.00 sec 1.17 MBytes 9.82 Mbits/sec
[ 9] 15.00-16.00 sec 1.48 MBytes 12.5 Mbits/sec
[ 11] 15.00-16.00 sec 1.40 MBytes 11.8 Mbits/sec
[ 13] 15.00-16.00 sec 1.62 MBytes 13.6 Mbits/sec
[ 15] 15.00-16.00 sec 1.64 MBytes 13.7 Mbits/sec
[SUM] 15.00-16.00 sec 8.19 MBytes 68.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 16.00-17.00 sec 1.08 MBytes 9.02 Mbits/sec
[ 7] 16.00-17.00 sec 1.23 MBytes 10.3 Mbits/sec
[ 9] 16.00-17.00 sec 1.59 MBytes 13.3 Mbits/sec
[ 11] 16.00-17.00 sec 1.48 MBytes 12.5 Mbits/sec
[ 13] 16.00-17.00 sec 1.83 MBytes 15.4 Mbits/sec
[ 15] 16.00-17.00 sec 1.61 MBytes 13.5 Mbits/sec
[SUM] 16.00-17.00 sec 8.82 MBytes 74.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 17.00-18.00 sec 1.39 MBytes 11.7 Mbits/sec
[ 7] 17.00-18.00 sec 976 KBytes 8.00 Mbits/sec
[ 9] 17.00-18.00 sec 1.60 MBytes 13.4 Mbits/sec
[ 11] 17.00-18.00 sec 1.18 MBytes 9.89 Mbits/sec
[ 13] 17.00-18.00 sec 1.82 MBytes 15.3 Mbits/sec
[ 15] 17.00-18.00 sec 1.14 MBytes 9.59 Mbits/sec
[SUM] 17.00-18.00 sec 8.09 MBytes 67.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 18.00-19.00 sec 1.61 MBytes 13.5 Mbits/sec
[ 7] 18.00-19.00 sec 1.11 MBytes 9.32 Mbits/sec
[ 9] 18.00-19.00 sec 1.70 MBytes 14.3 Mbits/sec
[ 11] 18.00-19.00 sec 1.27 MBytes 10.7 Mbits/sec
[ 13] 18.00-19.00 sec 1.14 MBytes 9.57 Mbits/sec
[ 15] 18.00-19.00 sec 1.00 MBytes 8.41 Mbits/sec
[SUM] 18.00-19.00 sec 7.84 MBytes 65.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 19.00-20.00 sec 1.12 MBytes 9.41 Mbits/sec
[ 7] 19.00-20.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 9] 19.00-20.00 sec 1.93 MBytes 16.2 Mbits/sec
[ 11] 19.00-20.00 sec 1.28 MBytes 10.7 Mbits/sec
[ 13] 19.00-20.00 sec 1.12 MBytes 9.37 Mbits/sec
[ 15] 19.00-20.00 sec 1.34 MBytes 11.2 Mbits/sec
[SUM] 19.00-20.00 sec 8.03 MBytes 67.3 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.03 sec 26.0 MBytes 10.9 Mbits/sec 72 sender
[ 5] 0.00-20.00 sec 25.3 MBytes 10.6 Mbits/sec receiver
[ 7] 0.00-20.03 sec 36.5 MBytes 15.3 Mbits/sec 82 sender
[ 7] 0.00-20.00 sec 35.8 MBytes 15.0 Mbits/sec receiver
[ 9] 0.00-20.03 sec 29.8 MBytes 12.5 Mbits/sec 25 sender
[ 9] 0.00-20.00 sec 29.4 MBytes 12.4 Mbits/sec receiver
[ 11] 0.00-20.03 sec 26.6 MBytes 11.1 Mbits/sec 34 sender
[ 11] 0.00-20.00 sec 26.2 MBytes 11.0 Mbits/sec receiver
[ 13] 0.00-20.03 sec 30.1 MBytes 12.6 Mbits/sec 32 sender
[ 13] 0.00-20.00 sec 29.8 MBytes 12.5 Mbits/sec receiver
[ 15] 0.00-20.03 sec 26.7 MBytes 11.2 Mbits/sec 30 sender
[ 15] 0.00-20.00 sec 26.2 MBytes 11.0 Mbits/sec receiver
[SUM] 0.00-20.03 sec 176 MBytes 73.6 Mbits/sec 275 sender
[SUM] 0.00-20.00 sec 173 MBytes 72.5 Mbits/sec receiver
iperf Done.
yeah that iperf pattern (one stream spikes then dies, needs parallel) is the same shape as the replication ceiling.
tailscale single-stream over cgnat often behaves like that even when the wan numbers look big on paper. try iperf3 -P 4 both ways and compare to a direct non-tailscale path if you can. if only the tailscale path is weak, the zfs send is just stuck behind that.
on the nas side keep an eye on whether the dataset has a ton of small files/records. one fat zvol or a dataset with big sequential files will look way healthier than a tree of tiny stuff at the same wan rate.
Now iperf results get weird: My PC can upload to WAN much faster than the truenas system in the same LAN.
- my PC to truenas server (same LAN): 6Gbit both directions
- my PC to iperf testserver (wan1): 330Mbit download, 180Mbit upload
- my PC to iperf testserver -P 4 flag: 570Mbit download, 480Mbit upload
- truenas server to iperf testserver (wan1): 330Mbit download, but only <50Mbit upload
- truenas server to iperf testserver -P 4 flag: 650Mbit download, but only 120Mbit upload
- truenas2 backup target to iperf testserver (wan2): 25Mbit upload, 85Mbit download
For this test run, tailscale was stopped. ping to iperf test server is the same from both systems, as is the number of hops.
Why can my desktop PC upload to the iperf test server with 3-4 times the speed of the truenas system? Both computers are literally connected to the same switch.
