Spontaneous massive decrease in write speeds/inbound speeds to NAS

TrueNAS General
, ,
1

I stumbled into a weird scenario, as of today, my write speeds from any PC on my network to my TrueNAS box has gone from ~150~250MB/sec down to around 2~4MB/sec for writes only. If I do reads from the server, everything basically maxes out at whatever the spinning discs can handle serving me. Yesterday everything was fine. I didn’t change anything on the NAS and it had been up and running since a recent reslivering 10-ish days ago. Not sure if it’s some phantom SMB quirk or what. Tried rebooting both the TrueNAS and my PC to no avail – both still report a 10Gbps full duplex connection but something seems to be clogging up or slowing down inbound writes.

The hardware of the NAS is :

cpu: Ryzen 5600G

ram: 2x32GB DDR4-3200

boot: 128GB random SilconPower nvme SSD boot drive

storage: 4x14TB HDDs (2 Seagate Exos X16s and then 2 WDs – a WD140EDFZ and a DC HC530 ~weird mix I know) in a 2x2 mirror configuration @51% Full

network: 10GbE TP-Link - AQC107 NIC passing thru a VirtualBridge

And here’s a local iperf run I just did:

Server listening on 5201 (test #1)

Accepted connection from 192.168.1.159, port 51578
[  5] local 192.168.1.160 port 5201 connected to 192.168.1.159 port 51579
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.21 MBytes  18.5 Mbits/sec
[  5]   1.00-2.00   sec  2.96 MBytes  24.9 Mbits/sec
[  5]   2.00-3.00   sec  2.80 MBytes  23.5 Mbits/sec
[  5]   3.00-4.00   sec  1.94 MBytes  16.3 Mbits/sec
[  5]   4.00-5.00   sec  1.38 MBytes  11.5 Mbits/sec
[  5]   5.00-6.00   sec  2.34 MBytes  19.7 Mbits/sec
[  5]   6.00-7.00   sec  3.99 MBytes  33.4 Mbits/sec
[  5]   7.00-8.00   sec  3.16 MBytes  26.5 Mbits/sec
[  5]   8.00-9.00   sec  3.73 MBytes  31.3 Mbits/sec
[  5]   9.00-10.00  sec  3.90 MBytes  32.7 Mbits/sec
[  5]  10.00-10.01  sec  32.8 KBytes  53.1 Mbits/sec




[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  28.4 MBytes  23.8 Mbits/sec                  receiver

Server listening on 5201 (test #2)

admin@truenas[~]$ iperf3 -c 192.168.1.159
Connecting to host 192.168.1.159, port 5201
[  5] local 192.168.1.160 port 34390 connected to 192.168.1.159 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.08 GBytes  9.25 Gbits/sec    5   1.24 MBytes
[  5]   1.00-2.00   sec  1.07 GBytes  9.22 Gbits/sec   15    953 KBytes
[  5]   2.00-3.00   sec  1.07 GBytes  9.23 Gbits/sec   19    913 KBytes
[  5]   3.00-4.00   sec  1.07 GBytes  9.22 Gbits/sec   21    665 KBytes
[  5]   4.00-5.00   sec  1.07 GBytes  9.23 Gbits/sec   13    915 KBytes
[  5]   5.00-6.00   sec  1.07 GBytes  9.22 Gbits/sec   20    926 KBytes
[  5]   6.00-7.00   sec  1.07 GBytes  9.23 Gbits/sec   21    915 KBytes
[  5]   7.00-8.00   sec  1.07 GBytes  9.22 Gbits/sec   27    952 KBytes
[  5]   8.00-9.00   sec  1.07 GBytes  9.23 Gbits/sec   26    727 KBytes
[  5]   9.00-10.00  sec  1.07 GBytes  9.22 Gbits/sec   32    670 KBytes




[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.7 GBytes  9.22 Gbits/sec  199             sender
[  5]   0.00-10.00  sec  10.7 GBytes  9.22 Gbits/sec                  receiver

iperf Done.

External tests are largely similar; inbound speeds are <10Mbps on the TrueNAS, but outbound speed to the remote ip is maxed. Limited by my 1Gbps WAN connection. iperfing to the same external test server on my PC is fully 1Gbps up/down.

Connecting to host 148.230.59.38, port 30000
[  5] local 192.168.1.160 port 55314 connected to 148.230.59.38 port 30000
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   109 MBytes   915 Mbits/sec   18    489 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   937 Mbits/sec    0    642 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   944 Mbits/sec    1    546 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec    0    686 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   944 Mbits/sec    1    594 KBytes       
[  5]   5.00-6.00   sec   111 MBytes   933 Mbits/sec    0    724 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   944 Mbits/sec    3    628 KBytes       
[  5]   7.00-8.00   sec   111 MBytes   933 Mbits/sec    0    752 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   944 Mbits/sec    3    662 KBytes       
[  5]   9.00-10.00  sec   111 MBytes   933 Mbits/sec    2    560 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec   28             sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  receiver

iperf Done.
admin@truenas[~]$ iperf3 -c 148.230.59.38 -R -p 30000 
Connecting to host 148.230.59.38, port 30000
Reverse mode, remote host 148.230.59.38 is sending
[  5] local 192.168.1.160 port 43378 connected to 148.230.59.38 port 30000
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   906 KBytes  7.42 Mbits/sec                  
[  5]   1.00-2.00   sec   874 KBytes  7.16 Mbits/sec                  
[  5]   2.00-3.00   sec  1.03 MBytes  8.68 Mbits/sec                  
[  5]   3.00-4.00   sec   690 KBytes  5.65 Mbits/sec                  
[  5]   4.00-5.00   sec   526 KBytes  4.31 Mbits/sec                  
[  5]   5.00-6.00   sec   496 KBytes  4.07 Mbits/sec                  
[  5]   6.00-7.00   sec   529 KBytes  4.33 Mbits/sec                  
[  5]   7.00-8.00   sec   619 KBytes  5.07 Mbits/sec                  
[  5]   8.00-9.00   sec   725 KBytes  5.94 Mbits/sec                  
[  5]   9.00-10.00  sec  1.29 MBytes  10.8 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.75 MBytes  6.50 Mbits/sec  430             sender
[  5]   0.00-10.00  sec  7.56 MBytes  6.34 Mbits/sec                  receiver

I’m really scratching my head here… if there are any other places to check to see what might be limiting inbound speeds suddenly or what.

2

I would check the connection at both ends of the TrueNAS NIC or try with a different cable.

If this was a disk or SMB issue, you wouldn’t see such abysmal speeds with iperf3.

Fun fact. This happened to me recently, but with a 1-GbE connection on my TrueNAS server. The connection from TrueNAS to the switch was loose. Forcing it to be tight by wedging a toothpick temporarily resolved the issue. I have since replaced the cable.

1 Like
3

You misconfiguration your networking? ports aren’t aligning? Acccepted is 51578 to 51579 for 192.168.1.159

Accepted connection from 192.168.1.159, port 51578
[  5] local 192.168.1.160 port 5201 connected to 192.168.1.159 port 51579
4

It was seemingly the cables. At least for now. Haha. I guess I should’ve tried that first.