Slow transfer speeds over 10G when migrating from old NAS to new

TrueNAS General
,
1

Hi all! I’m new to these forums, but not new to the old FreeNAS community.

I’m experiencing slow transfer speeds (~18-20 MiB/s) when migrating data from my old NAS (FreeNAS 11.3-U5, specs below) to the new one (TrueNAS CORE 11.3-U6, also below). Both are custom, bare-metal builds. Total data is around 28 TiB across a half-dozen datasets. This has honestly been an ongoing problem with downloading anything from the old NAS that I’ve been embarrassingly too lazy/busy to fix, but now it’s a big issue.

Transferring 28 TiB at 18 MiB/s would take about 18 days, when it should take about 8 hours. Both machines are connected via copper to a 10G Unifi aggregation switch, yet I’m not even getting gigabit speeds. I am able to confirm bidirectional ~10G speeds using iperf3. However, when I transfer anything from the old NAS by sshing into it and using rsync over ssh to the new server (to transfer as directly as possible), it just drags.

The pool on the originating NAS is GELI encrypted. The new pool is ZFS encrypted. I expect encryption to cause a performance hit, but nothing quite this drastic.

Happy to answer any and all questions. Thank you very much in advance. Here are the specs and some screenshots.

“NAS” 10.10.20.10

Supermicro X9SRH-7F (w/ onboard LSI 2308 HBA)
Intel Xeon E5-2628L v2 1.9GHz 8c/16t LGA2011 SR1AF
2x 16 GB DDR3 PC3-12800 RDIMMs
Boot: 2x Transcend SSD370S 32 GB SSDs in RAID1
vdev: 8x HGST Deskstar NAS 6 TB (48 TB raw, 32 TiB usable, raidz2)
Seasonic 80+ gold 550W PSU
2x Athena Power 5 in 3 HDD cage/backplane
iStarUSA D-400-6 chassis
FreeNAS-11.3-U5

“NAS2” 10.10.20.11

Supermicro X11SSH-TF
Supermicro 9300-8i SAS3 HBA
Intel Xeon E3-1270 v6 SR326 3.80GHz 4c/8t LGA1151
4x 16GB DDR4 PC4-2666 2RX8 UDIMMs
Boot: 2x Transcend SSD370S 32 GB SSDs in RAID1
vdev: 6x (+1 hot spare) Seagate Exos X20 20TB - ST20000NM007D (120 TB raw, 72 TiB usable, raidz2)
l2arc: Samsung 970 Evo Plus 1 TB M.2-2280 PCIe 3.0 X4 NVME
Supermicro SE826 2U Server 12 Caddy Chassis w/ redundant PSUs
TrueNAS CORE 11.3-U6

iperf_nas
iperf_nas731×113 2.8 KB

slow_nas_speed
slow_nas_speed2

2

I suspect this is at least partly due to the utilization (86%) of the old NAS pool, hence the desire to migrate to a larger server.

I’m considering doing a bit of “sneakernet” and just moving the stuff over in batches via an external drive. Even at USB2 speeds, that’d still be way faster than this slog.

Oddly, the transfer speeds are asymmetric. When copying from the new NAS to the old, the speeds are doubled (38 vs. 19 MB/s). Still terrible, though.

root@truenas[~]# rsync -avz --info=progress2 -e ssh /mnt/path/to/file root@nas.mydomain.com:/mnt/path/to/file
root@nas.mydomain.com's password:
sending incremental file list
file
  2,166,751,232  40%   38.03MB/s    0:01:22

What I’m seeking are ideas as to what might be causing both the discrepancy and the overall slowness. I get that spinning platters will not saturate a 10G connection, but it should at least be somewhere to the tune of 100 MB/s with a single-digit-percentage hit for encryption.

EDIT: I deleted the post below because I was too quick to dismiss it as only being utilization. The linked thread in the deleted post goes into more detail and is VERY similar to what I’m experience. Unlike that thread, though, my two machines are on the same VLAN and on the same 10G switch, transfers are not going through the router, and iperf is reporting normal transfer speeds. Something else might be going on.