Extremely poor iSCSI performance | SSDs | Proxmox

TrueNAS General
,
22

my allocation unit size i have set is 2048k i use the share for video editing. mostly

23

On the zvol?

24

I set it through windows. Using the disk management program built into windows.
I could not find an easy way to change the record size in truenas other than deleting and recreating the zvol.

On that if anyone knows if it inherits that from its parent dataset. that would be cool to know

25

Hello, I’m back.

I changed data VDEV to:
1 x RAIDZ1 | 4 wide | 1.82 TiB (Usable Capacity: 5.15 TiB)
(not as solution, I want more space)

I finally replaced the network card “Asus XG-C100C V2” (w11 side) with an “Intel X550-T2”.
And configured “Intel X550-T2” as follows:

1. Interrupt moderation: Disabled ("low" is OK as well)
2. Jumbo packet: 9014
3. Receive buffers: 2048
4. Transmit buffers: 2048

Connection scheme is the same (direct wired connection):
w11: Intel X550-T2 → RJ45 → Ubiquiti RJ45 to SFP+ → proxmox: Intel x520-DA2
As before, TrueNAS directly uses a dedicated port from the Intel x520-DA2 (PCI Passthrough).

Here are the results:
SMB speed as fast as before, no changes - 1.12 GB/s

iSCSI:

iSCSI Zvol:

Type: VOLUME
Sync: DISABLED
Compression Level: OFF
ZFS Deduplication: OFF
Volblocksize: 64K

iscsi-vol64k-sec4k

iSCSI Zvol:

Type: VOLUME
Sync: DISABLED
Compression Level: OFF
ZFS Deduplication: OFF
Volblocksize: 128K

iscsi-vol128k-sec4k.png

------------------------------------------------------------------------------
CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):  1217.902 MB/s [   1161.5 IOPS] <  6878.02 us>
  SEQ    1MiB (Q=  1, T= 1):   878.151 MB/s [    837.5 IOPS] <  1193.70 us>
  RND    4KiB (Q= 32, T= 1):   369.527 MB/s [  90216.6 IOPS] <   353.52 us>
  RND    4KiB (Q=  1, T= 1):    41.126 MB/s [  10040.5 IOPS] <    99.49 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):   945.361 MB/s [    901.6 IOPS] <  8853.96 us>
  SEQ    1MiB (Q=  1, T= 1):   972.868 MB/s [    927.8 IOPS] <  1077.29 us>
  RND    4KiB (Q= 32, T= 1):   119.451 MB/s [  29162.8 IOPS] <  1096.45 us>
  RND    4KiB (Q=  1, T= 1):    35.311 MB/s [   8620.8 IOPS] <   115.88 us>

Profile: Default
   Test: 2 GiB (x1) [F: 0% (0/200GiB)]
   Mode: [Admin]
   Time: Measure 5 sec / Interval 5 sec 
     OS: Windows 11 Professional [10.0 Build 22631] (x64)

This is much faster than it was before, although I think the hardware is capable of reaching 1.2-1.4 GB/s with 4x sata SSD @ RAIDZ1 (based on SMB multipath test with 2 x 10Gbps channels).
I will also test Synology ds923+ iSCSI speed with the same SATA SSDs.

So, I think in my case, the solution is:

  1. Use server grade NICs on both sides.
  2. iSCSI Zvol Volblocksize should be: 64K
  3. iSCSI Extent Logical Block Size should be 4K

General advises:

  1. Configure NICs for low latency (Interrupt moderation)
  2. Don’t forget to set proper Jumbo packet size.
  3. Increase Receive and Transmit buffers (2048 should be OK).
  4. Use a direct wired connection or the same network subnet to avoid routing traffic through the router or consider very powerful router (I bet that UXG-Enterprise or EFG should do the trick).
  5. Sync=Disabled increases the write speed, but in case of system failure there will be data loss for 5 seconds.
  6. Make sure that write caching is enabled on your disks (see this topic for details).
3 Likes
26

I’m curious which of the above actually fixed the problem. My TrueNAS VM is dual-role; it provides iSCSI to its own all-in-one ESXi host and it also provides iSCSI to my Windows 11 desktop via 10Gb Ethernet.

The latter has always moved data within 10% of “full line rate” no matter what I do – any NIC setting on either end of the wire, any volblocksize (within reason), any ZFS recordsize (within reason), any NTFS allocation unit size (within reason). I say “within reason” because I’ve never come across a setup where someone used NTFS w/ 2M allocation units and I’ve no idea if it’s a factor here. In my experience in the enterprise world long sequential I/O that warrants deviation from the default 4k always gets 64k. Are there setups in the wild that profit from >64k?

27

thanks. looking at doing this myself, so this is great. thank you for posting.

28

Setting sync to disabled is not recommended for production workloads. I’m explicitly stating this because some users may uncritically apply your recommendations and end up with data corruption.

2 Likes