Slow nfs on truenas 25.04.2.4 CE

TrueNAS General
1

Hi,

I have a new and rather beefy machine built for Truenas CE to serve NFS

Supermicro AS-2015CS-TNR
AMD EPYC 9124 16-Core Processor
384 GB RAM
16 x WD HC550 WUH721818AL5204 (2 vdev)
2 x Micro 7450 NVMeTM SSD (slog)
configured as
Data VDEV: 2 x RAIDZ2 | 6 wide | 16.37 TiB
Log VDEV: 1 x MIRROR | 2 wide | 745.21 GiB

Networking : 1 Gb and 100 Gb Infiniband. I am configuring Truenas to do IPoverIB but nothing fancy. Note: the performance issue is seen over 1Gb ethernet too.

From the client, a simple dd write to the NFS mount, this sometimes will be low as 5-30 MB/s despite the storage only streaming about 1 Gb for backup data --read only.

dd if=/dev/zero of=testfile1 bs=1M count=1024 oflag=direct

…

This dd same command ran from the storage itself sees over 5.5 GB/s. The load average on the truenas hovers around 10 – which seems pretty high for only stream 1 Gb (gigabit) of data.

Is there anything to improve the write performance? The backup / management is limited intentionally to 1 Gb so that the system stays underutilized.

2

What mount options did you use on the client?

3

rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys

tried nfs 4.1 and 4.0 along with smaller block size. nfsd appears to auto set truenas side with 16 threads.

4

What recordsize is the dataset using?

Can you confirm the SLOG is being used?

5

ill try to confirm. recordsize is default 128k

6

Change it to 1M and see if there is any difference in performance with new writes.

Compression should be set to LZ4 at minimum.

7

am already using lz4 and changing to 1M didnt really improve the write performance. Oddly enough I canceled the backup job that was streaming 1Gb (read) and that didnt help either.

8

note: I was finally able to restore write performance by killing the 1 Gb backup job. It just seems it can do read, write but not both together.

image
image1465×419 31.3 KB

After I killed the job, that same DD command writes data at ~1 Gigabyte/sec. Its just having a single read job at 1 Gigabit completely tanks write performance.

9

When the backup job was restarted it murders the performance of writes. From ~1.1 GB (for the dd) down to 23 MB/s.

Such a strange issue.