I’ve set up a new fresh install of TrueNAS Scale 24.04.2. 96GB RAM, two 22TB CMR HDDs in a mirror, for testing.
The server and the client are connected to a 10G switch each. I ran iperf3 in both directions to confirm that that works (9.x GBit/s in each direction).
I mounted the share on my Linux box (tested Ubuntu 20.04 and Debian 12) with the following options (NFSv4):
“proto=tcp6,soft,async,rsize=1048576,wsize=1048576,timeo=600”
(I also tested NFSv3 and/or smaller rsize/wsize parameters).
The share is basically empty, there’s just a couple test files or folders. However, I have the following issue:
- When I mount the share with the “async” option, then I get great write performance, up to the maximum supported by my drive. However, when I have one process writing to the share at full speed (like, just a “cat /dev/zero > file.bin”) and then try to read anything from the share with another process, that doesn’t work. A simple “ls” call in another terminal window is stuck and doesn’t return (I’ve let it sit for multiple minutes!) until I cancel the writing process, at which point the “ls” immediately returns.
- When I mount the share with the “sync” option instead, the issue is gone - I can write with one process and at the same time list a directory or read files in another. The issue is, the speeds are sloooow. I know that “sync” is slower because it has to flush every write to disk first, but I’m talking about write speeds of under one MB/s - that can’t be normal either, can it?
I’m fine with the “async” mount not being 100% safe in case the machine crashes, I’m “just” storing data on the share, nothing like VM images or databases. But I need to be able to at least navigate folders and open small files while another process is writing data …
I searched around on Google and found a ton of NFS speed optimization guides, but nothing that really applied to my issue. I found a few forum posts like this one from years ago where someone had the exact same issue, but I found no solution.
The drives and the network are fine; when I mount the share using SMB instead of NFSv3/v4 I get great read/write speeds and can write and read at the same time; but I can’t use SMB for my data because TrueNAS still doesn’t support SMB3 POSIX extensions so I can have proper file ownership and chmod bits for my files.
Is there anything else I can do / test / change to get NFS to work properly? (Or is there any plans when TrueNAS will support SMB3 unix extensions?) I can’t imagine that this sort of behaviour is normal so there must be some issue in my configuration or environment - but it’s a fresh install …