Is it possible to connect your NAS directly to your Windows machine (workstation)
My test was NAS with Intel X520 connected with fiber to Window 10 or 11 with Intel X520. My SFP+ connect using LC connectors and MMF.
With you having different speeds of NICs and switch(es), what are your connection options? It isn’t really clear what the network path is when you test.
Yes, I tried that one or two hours ago, as described above.
With a direct connection using on Windows a 2 port 40G card (one to the NAS, the other to the global switch) and a 9000 MTU, I get twice the speed for large file transfers.
To clarify:
My global network is a 10G one with a Netgear 10G switch.
I was testing initially by using 100G cards on the PC and the NAS, both connected to a 100G Mikrotik switch. The Mikrotik acts as a relay to the global 10G network using a DAC breakout cable connected to the Netgear.
With this configuration, I get barely 10G speed using a (PC → Mikrotik → NAS) 100G connection. Around 600 to 800 MBs for a large file transfer.
I just tried:
The raw RAID speed runs at 3.8GBs, so even with a direct connection, the network is limiting, but less.
Are we still talking about your 18 wide RAIDZ3 VDEV?
Btw the way, do you have more than a single NIC port connected to your server as you are running these tests? If not, try the iperf3 with only one network cable attached.
Did you try to add a slog vdev (ZFS LOG)?
SLOG is helpful for synchronous writes
See Hybrid Storage & Flash Cache (SLOG/ZIL/L2ARC) section in Hardware Guide of current Scale version.
I’m not sure. I know I tried a metadata vdev and that improvement was marginal.
But slog might only improve the vdev raw speed for sync writes, and SMB (CIFS) typically performs asynchronous writes.
The internal write speed is already already at 3.8GBs as shown wit the fio test I ran.
The bottleneck is the network speed.
Sorry but thats a rather not-helpful piece of advice.
If he wants to test if its sync related, easiest way to test is just disable sync for testing purposes
There are some things to take into account:
Some nic rely on the CPU, with multi-cpus that might create issues if cores on multiple CPUs are used.
Connect-x2 shoiuldnt be used, it heavily outdated.
Check your PCIe lanes and which CPUs connect to which PCIe lanes.
Exclude the switch
sorry but it’s two test in one: is it sync related AND speed benefit of a slog. And your don’t give advice but things not to do with no things to do! (and judging others…)
You can test the same thing by disabling Sync.
Most users do not need sync writes so putting in a slog is a waste and requires hardware changes.
I do give advice, I just wrote 4 things to check and look out for.
And yes, I judge others if they give bad advice or create weird/non-standard setups.
ConnectX-2 is just used now to serve the global 10G network in my house.
On TrueNAS, my final configuration now is to use a dual port 40G ConnectX-3.
That goes directly to my 2 PCs that need the faster possible connection.
It turns out that ConnectX-3 does not work well with Windows 10 22H2: the second port cannot link. Not such problem on TrueNAS.
One of my PCs has only one free PCI slot, so I had to use a dual port ConnectX-4 card, a 56G that I borrowed to a friend. I’ll replace it with a 100G that costs 8 Euros more on eBay.
The second PC does have free PCI sots, so it’s using 2 cards, one being a 100G connected to the TrueNAS.
Here are the results, similar on both PCs:
Raw internal RAID write speed it 3800MBs, so the network is still limiting, but much less than before. It’s already respectable.
ConnectX-2
Will likely not get great speeds in most cases, be watchful of that.
ConnectX-3 or 4 (4 is pretty nicely priced now) is where its at.
ConnectX-3
I’m pretty sure our card is broken, as I’ve been able to run that on any version of win 10 in the past.
a 56G that I borrowed to a friend
Thats not a eth card if it says 56g, likely infiniband or dual purpose.
Check the firmware!
Here are the results, similar on both PCs:
Those 4k q1t1 make me cry, try to (re)add metadata vdev (MIRRROR!) with small blocks on 4k or 8k.
Think about using 3x6 raidz2 instead, its statistically safer at minor cost and about 3x as fast when it comes to iops
On another note:
There are a few more, I’ll send later… But currently not the time!
I thought so too at first, but I tested 3 cards, all behave the same.
Yes, these cards are odd ducks.
Using "mlxconfig -d mt4115_pciconf0 q " shows that they are in Ethernet mode.
The Mikrotik does not offer a 56G setting, so I run it at 40G anyway.
I agree on the principle, but I’ll use the NAS mostly for large video files, making copies or remuxing them, either from itself or from a Gen 5 or 4 NVME SSD.
So what matters for me if sequential write and read speed, with at most 2 or 3 concurrent tasks.
Here is what I get with 2 PCs running the same test at the same time:
That’s an impressive 3.5G read speed and 1.8G write speed.
iops just do not matter for my usage, and with 6 parity drives total instead of 3, I would loose 66TB!
I set compression OFF, with video files it’s counter productive.
I’m not sure about knowing if a dataset needs sync or not? Is it related for example to using an UPS so that there won’t be unexpected shutdowns?
I tried to set it off already, and the benefits were marginal with sequential R/W speed, so I prefer to leave it on.
atime is off.
No hurry, I really appreciate your input, that makes me learn things. The iops optimization is interesting to know even though it’s not my concern for this particular setup.
Thanks!
Or so you think… But if you ever edit video files on the HDD pool you’ll need IOPS.
At the very least, two 9-wide raidz2 would be better than a single 18-wide raidz3.
No raw vido files? The cost of checking that files are indeed incompressible is negligible; you really should enable lz4 or zstd-fast.
For transactions which cannot be replayed after a crash. Think databases, accounting, trading… Or block storage (VM, iSCSI), where the host fle system could get corrupted.
For video editing on SMB shares, disable sync writes.
lz4 it is.
Next time, I’ll try 2 RAIDZ2 instead, but I already uploaded all my files, capacity is at 60%, I am not looking forward to redo it!
Edit: I just remembered that I did try it. At the time, I did not have a direct link like I do now, and it did not change the less fast sequential R/W speed at all.
I don’t edit video files, just remux them after fixing subs for example, or making an MKV from a Blu-ray I ripped.
I’ll disable sync writes, and see how much it changes muxing time for example.
Edit: disabling sync boosts sequential read by 12% and leaves sequential write unchanged (with CrystalDiskMark). I would have guessed the opposite!
Or so you think… But if you ever edit video files on the HDD pool you’ll need IOPS.
Also metadata on SSD and a tad more tops is also nice because file browsing and such.
and if you have a metadata vdev already, setting smallblocks to 4K (the smallest option) is a great idea. There just wont be many 4K blocks with video files, so not will al handle itself.
But yes, Iops do mater, even for things like video scrolling.
you really should enable lz4 or zstd-fast.
Interesting tidbit: the current implementation of ZSTD, actually does the first compressability checking using LZ4. ZSTD-1 is pretty comparable with LZ4 in performance btw.
However, for video files I would indeed just use LZ4. It has almost zero downside.
lz4 it is
Yeah that objectively is the way, for 99,999% of use cases.
Next time, I’ll try 2 RAIDZ2 instead, but I already uploaded all my files, capacity is at 60%, I am not looking forward to redo it!
Next time dont do such things before you’re happy with the system beforehand.
Some more tips:
Recordsize, if you just remux and transcode… Go high, 1M or 2M is not silly.
If you put 4K or 8K smallblacks on the metadata vdev you already added, that shouldn’t give any noticeable downsides.
xattr you want on “sa”, the default is just for backwards compatibility.
Do note that all those changes are ONLY applied when you add data to a dataset.
Thats why you need to tweak things first and then add data or use a script to rewrite all data.
That’s a little bit condescending.
I was happy in fact, I made extensive tests. If you reread my posts, I did try 2 vdevs instead of one, and found no real benefit for large file transfers. You’re fixated on iops, that does not apply to my setup where I prefer more capacity.
it’s the network side that’s not 100%. None of your suggestions helped for that.
Enabling compression is a good idea since there is no downside, but it does not change anything for 99% of the data, nor help with network speed.
The only thing that helped is 1M block size, which I already did. Higher gives less speed. And using a direct connection with a 9000 MTU, which can be done after filling up.
I did not need you for that.
It isn’t its just a statement of fact for all users, not to fill systems before being fully happy with them.
it’s the network side that’s not 100%
I’m not 100% certain thats the case, it could just as well be transfer protocol related.
None of your suggestions helped for that.
I prefer to work from ZFS downwards, I wasn’t even talking or looking at network transfer optimisation yet…
Higher gives less speed
Yes, there is some funkiness with running the newer record sizes higher than 2M… It’s interesting to try though.
I did not need you for that.
I wish you the best, I’ll leave you at it.
