I have read what feels like hundreds of posts on this issue, so I am sorry for creating another one, but hopefully I can answer a bunch of questions up front. Issue: I am getting 55MB/s via Windows file copy over SMB (copying a 20GB video file).
I have a synology NAS that I saturate its 1gig connection (110MB/s) over SMB with the same file.
I have run iperf tests and dd speed tests on my ZFS pool (using /dev/random and making sure compression was disabled)
My pool is 4x 22TB Seagate Exos HDDs connected via a LSI 9600-16i with PCIe passthrough to my truenas scale VM running in proxmox. I have given TrueNAS 8 cores and 128GB of RAM. (Running an AMD Epyc Genoa 9334 32 core CPU with 256GB RAM on the host) Finally, the proxmox host has an Intel x710 with a SFP+ fiber connection to my 10-gig switch which connects over another 10-gig connection to another switch. Finally I connect to my desktop at 2.5gbps.
With all the hardware above, I was expected to AT LEAST get gigabit speeds, as based off the iperf test and dd zpool test I should be able to easily hit 100+ MB/s
It always better to performance test first with:
- Writing a file
- Reading a file
- then copying a file
Copying is often much slower due to software limitations (like low queue depth)
Don’t use dd to test read/write speeds, use fio - google it up
Make sure you run fio from a dataset on your zfs pool
You to check/do these things :
Check MTU size and set to 9000 on NAS, 9014 on Windows
Check whether SMB multichannel is enabled on both TrueNAS and Windows sides.
Set-SmbClientConfiguration -EnableMultiChannel $true
Check whether RSS is enabled on Windows
Get-NetAdapterRSS
And check that you see multiple connections on Windows to TrueNAS’s share :
Get-SmbConnection
Get-SmbMultichannelConnection
Thanks for the help bugacha! I started by re-doing my tests with fio. Below, you will find my write benchmark. I enabled sync=always to ensure I wasn’t testing RAM speeds.
I tried to perform read fio tests, but I got 78GB/s as the file is stored in RAM. If you absolutely need read tests, I can change the VM to have only 4GB of ram and do a read test with a 20GB file or something similar.
I know MTU size is default (1500?) so I can certainly change that, and enable SMB multichannel, however my synology NAS doesn’t have these set and I am able to get full gigabit speeds, so I am confused as to why I have to make special configurations on my windows client.
I will test these config changes and report back later today. Thanks again for your expertise.
you can leave MTU unchanged, but you deffo need to enable SMB multi-channel
I got SMB multichannel setup and working, but the speeds I get are still 45-50MB/s.
TrueNas 2 network adapters during file transfer:
My Windows 11 desktop (client) showing 2 connections:
Yeah, considering the low speeds here and the fact that you have a good CPU, multichannel was unlikely to have an impact.
The bottleneck is elsewhere.
Edit: Hmm, why does it report your client IP as 100.X.X.X? That’s a whole different subnet, and implies an ISP assigned IP, specifically a CGNAT IP.
Is this a routing issue?
Are you copying the file from your desktop’s local disk or from your synology or other network appliance?
I am copying the test file from my local nvme disk. I also tested my transfer speeds on a Mac laptop and got similar speeds.
This screenshot says nothing about whether SMB multi-channel works
you need to look at Get-SmbConnection
Also RSS = false is bad, are you using Virtual nics ?
I followed the Truenas documentation page for enabling and configuring SMB multichannel, along with the commands given for checking if its enabled. I also saw the traffic split across two network interfaces, which proved to me its working.
Here is the output:
And yes, TrueNas is virtualized in Proxmox. The host has a dual sfp+ Intel X710.
If the NIC that you give to TrueNAS is virtualized and not PCI-e pass through, make sure you add this to cli
service smb config
| smb_options | interfaces="192.168.1.3;speed=25000000000,capability=RSS" |
service smb update smb_options="interfaces=\"192.168.1.3;speed=25000000000,capability=RSS\""
adjust speed to your liking
also make sure that service smb config shows multichannel | true
also make sure you set Multiqueue to number of CPUs you give to TrueNAS in Proxmox
You dont need to have 2 NICs to have working SMB Multichannel btw
TrueNAS documentation is crap, they completely ignore the fact that working RSS allows multiple SMB connections on a single NIC.
I saw on the microsoft docs that with RSS a single NIC could do multichannel, but figured I would follow the TrueNas docs. I’ve gone ahead and removed the second nic now.
In the TrueNas VM, I set the samba config from your post. Used 10gig as the VM host is connected at 10gig speed. I also restarted the SMB service and rebooted truenas itself.
The RSS, on my desktop still shows as disabled. But this says client, ie my desktop, not truenas? I am confused as to what I need to change where to fix the RSS issue. I greatly appreciate your help debugging this, I think we’ve narrowed my original issue down to the culprit.
TrueNAS side is good now (just double check Multiqueue setting in Proxmox on virtual nic, it needs to be set to number of CPUs you give to truenas)
Yes, your client NIC needs to support RSS as well. There is usually a setting in the driver on Windows. Sorry thread is a bit long, I don’t remember is your client windows?
Oops, forgot to share the screenshot, set multiqueue to 8, as the truenas VM has 8 cores. To be safe, I restarted the VM as well.
Get-SmbMultichannelConnection | ConvertTo-Json reports currentChannels=4, previously it reported only 1.
I also discovered my 2.5g realtek nic doesnt support RSS, so I switched to a 1g Intel nic that does, and now it reports as RSS capable:
However, even with all this setup, I am still stuck at 50MB/s
CPUs that you provide into TrueNAS - are they set as host? This is very important to provide them as host and not emulated into something else.
For completeness, I would try MTU 9000 / 9014 (win) to see if speed changes.
Yes, the CPU is set as host. Here is my entire proxmox config in case its helpful:
I set the MTU to 9000 on my physical nic (via proxmox) updated the virtual bridge adapter in proxmox to 9000, and set the virtual adapter in the Truenas VM to 9000. In my desktop network properties, I enabled Jumbo Frames and set it to 9014 from the dropdown. Also enabled jumbo frame support on my switch.
Result: Still the same ~50MB/s transfer speeds. 
And just to check I am not insane: I tried copying the same test file from my desktop to my old synology nas, and I am getting full gigabit speeds.
Just to double check, did you do iperf3 test in the direction of copy? I think the one in the first post isn’t exactly correct.
Iperf3 should run as server on NAS, with Windows client connecting.
But yes, the fact that it caps out to 50 is very strange, something is off.
