Iperf3 with aqc113C, slow in one direction only, but only if the sender is Linux

TrueNAS General
1

Hello,

I noticed my backup NAS (TS-653D, with TrueNAS Scale 25.04.2.6) was appearing slower when writing my backup from the big NAS (same OS) than anticipated. The big NAS is 10000Base-T (10Gbe), the backup NAS has a PCIe-based aqc113C, linked up with 5000Base-T (5Gbe). Transfer speed to a pool with 6 spindles was only 1.5Gbps, whereas i know from testing, it manages to do 6-7Gbps sustained sequential writing (which is the type of operation i expect during backup as it is block based and the destination ZFS pool is not fragmented). 1.5Gbps instead of 4Gbps is a time difference for 14TB.

This led me to do some iperf3 (iperf3 -s, iperf3 -c -P4). Just showing the matrix:

Source / Destination →

MAC→backup NAS (4,91Gbps no retries OK)
Linux1 10Gbe→backup NAS (1,5-3Gbps 10k retries NOK)
Linux2 10Gbe→backup NAS (1,5-3Gbps 10k retries NOK)
big NAS 10Gbe→backup NAS (1,5-3Gbps 10k retries NOK)
backup NAS→all hosts 4,95Gbps no retries OK
all hosts 10Gbe→big NAS 10Gbe 9,9Gbps no retries OK

I played with rx ringbuffer size (currently 4096, but did not make a difference with hardware maximum of 8000). For the sake of the argument, i limited the MTU back to 1500 on the vlan adapter on big and backup NAS and checked again, different values, same problem (1Gbps). Funnily enough, the MAC with MTU9000 still sent 4,6Gbps to the backup NAS….
The MTU in my LAN is 9000 and the results above are taken using this value.

I noticed, when macOS sends, the backup NAS CPU (J4125) goes up to 22% for the test period, whereas all attempts with linux PCs just go up 18% and much shorter, just like a spike - is this any indication?

i would have liked to limit one of the Linux PCs to 5Gbps, but they are all fiber so missing the advertised mode.

My work hypothesis is that the aqc113C is able to RX 5Gbps fine (good case macOS sender) and it must be something from Linux side interfering with bitrate, maybe not really able to ratelimit traffic, so probably oversaturating the TCP connection to the slower interface, then retrying and repeat. I also estimate that this observation is the reason for the slow write rate of backup NAS.

Can anybody advise what to do? (I upgraded to 25.10.2 but same behavior)

(Update: i redid some tests from Linux hosts with iperf3 -c -P4 -u -b 5G and got the expected results with UDP. It seems to concentrate on TCP.)

2

Update: I was able to receive constant iperf3 4,95Gbps sending from Linux PCs after enabling SACK.

Update2: The transfer speed is also limited by the CPUs ssh performance which i could increase a little bit by using -c aes128-gcm@openssh.com (intel AES-NI extensions, ssh option). FYI, the speed is now 2,2Gbps which is better than nothing (with interface link still 2.5Gbps). One could also send in the clear.

Update3: By using unencrypted transfer, the speed could further be increased to 4Gbps using nc.

Update4: I have the htop running during the backup and the reporting network rx does not correlate with what the TrueNAS GUI shows as Out - therefore, and because the written data (iostat) is also much more than what is reported and because compressed data is involved, i assume that the compression should explain this.