iSCSI, scst, and process count

TrueNAS General
,
1

Can someone with knowledge about iSCSI and scst shed some light on the the massive number of threads that are created when sharing a iSCSI LUN.

There is a user who is upgrading from CORE to SCALE and has noticed the difference in operation.

In my case I have 6337 processes for 18 disks

root@nas[~]# ps x | grep VMFS1 | wc -l
6337

~ 18 disks * 32threads * 12cpu_threads

I’ve seen process counts in the 10s of thousands.

I’m assuming that this is normal because it’s default behavior. Are 32 treads per disk per cpu thread necessary? Is this tunable?

I’m not an iSCSI expert so could use some input.

2

to add a few more facts

hardware

root@nas-01 ~ # lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   12
  On-line CPU(s) list:    0-11
Vendor ID:                GenuineIntel
  BIOS Vendor ID:         Intel(R) Corporation
  Model name:             Intel(R) Xeon(R) E-2386G CPU @ 3.50GHz
....
root@nas-01 ~ # free -h
               total        used        free      shared  buff/cache   available
Mem:           125Gi       112Gi       7.7Gi        11Mi       6.9Gi        13Gi
Swap:             0B          0B          0B
root@nas-01 ~ # perl disk-infos.pl
Device        Vendor/product            Firmware Serial                    Capacity   Temp
/dev/sda      ATA      INTEL SSDSC2KB03 0120     BTYI402002293P8EGN        3840.76 GB (27 C)
/dev/sdb      ATA      INTEL SSDSC2KB03 0120     PHYI3433003W3P8EGN        3840.76 GB (27 C)
/dev/sdc      ATA      INTEL SSDSC2KB03 0120     BTYI402001ZJ3P8EGN        3840.76 GB (27 C)
/dev/sdd      ATA      INTEL SSDSC2KB03 0120     BTYI402001AZ3P8EGN        3840.76 GB (26 C)
/dev/sde      ATA      INTEL SSDSC2KB03 0120     BTYI402000TQ3P8EGN        3840.76 GB (27 C)
/dev/sdf      ATA      INTEL SSDSC2KB03 0120     BTYI402305JH3P8EGN        3840.76 GB (27 C)
/dev/sdg      ATA      INTEL SSDSC2KB03 0120     PHYI3433028U3P8EGN        3840.76 GB (27 C)
/dev/sdh      ATA      INTEL SSDSC2KB03 0120     PHYI343302AG3P8EGN        3840.76 GB (27 C)

And it is not like this would slow down the system so drastically that it is a burden (at least for now, lets see what will happen during the week)
But i still wonder if that amount is really necessary.

I could imagine those 32 threads were chosen for larger setups and not really worth it with one to one connections w/o any other systems simultaneously throwing their bits around.

now while i’m writing this i found the following.

scst’s readme - teaser

Additionally VDISK handler has module parameter “num_threads”, which
specifies count of I/O threads for each FILEIO VDISK’s or VCDROM device.
If you have a workload, which tends to produce rather random accesses
(e.g. DB-like), you should increase this count to a bigger value, like
32. If you have a rather sequential workload, you should decrease it to
a lower value, like number of CPUs on the target or even 1. Due to some
limitations of Linux I/O subsystem, increasing number of I/O threads too
much leads to sequential performance drop, especially with deadline
scheduler, so decreasing it can improve sequential performance. The
default provides a good compromise between random and sequential
accesses.

So i guess my assumption wasn’t so far from the truth

You shouldn’t be afraid to have too many VDISK I/O threads if you have
many VDISK devices. Kernel threads consume very little amount of
resources (several KBs) and only necessary threads will be used by SCST,
so the threads will not trash your system.

  • “Don’t Panic, keep Calm!”
    Yes, only certain things in the system are slow on “initialization” - basically anything which looks at the process table :see_no_evil:

As @BooMShanKerX already said, hopefully someone with a bit more insight could share some light into this and maybe this even leads to a new feature so people could tweak the scsi service a bit to to their individual needs.

Not everyone wants to use their nas as a hypervisor :sparkler:

3

My understanding is just parallelism… more threads = more processing power and better use of multi-core CPUs.

The only thing to worry about is CPU utilization for a given workload or what is the max workload?

4

Yes, this is a great thing. Not like so many things you do here and there like just “compressing” a file where you look at one core sticking it out and the rest is sleeping :—).
Maybe it’s normal like this (or at least with some iscsi implementations on linux). The last 20+ years i ran solaris, illumnos, openindiana, napp-it … and since i switched to true^WFREEnas with like 8.6 or so i only had ESXi’s talking to them over iSCSI. And all the systems i look after are usually happy to do the CIFS + NFS shares at the same time (not that this would have any big impact unless there were some power users / services which then get their dedicated pools).

So just “feeling” those lags because i wanted to look at the processes to see if the replication was running or starting a htop i wondered wtf is now going on.
And running a ps spewed out so many lines that even my 10k buffer in tmux wasn’t able to cope with it. I already get annoyed on systems which just throw you ~2-3k lines as the have so many core whicht adds up…
This just made me wonder if something is wrong here or if this was just the normal behaviours anyone is ok with (and why i asked @BooMShanKerX if they maybe had similiar experiences).

As this Scale system is only a day old i will know in a couple days. I mostly ran backups on Proxmox now to get some traffic going. And it’s not like that this thing would have to do that much anyhow. It’s a small setup. This was the first Proxmox System here which someone else “planed” and because of that i have to use iSCSI. On newer systems i just put the zpools into proxmox so i won’t have to bother with this.

1 Like