Special VDEV (sVDEV) Planning, Sizing, and Considerations

Drop Kris and I a line - t3@truenas.com - and we’ll see if we can do a viewer-questions heavy show. There’s several caveats around sVDEV use (as this thread’s existence speaks to!) and using it safely.

Didn’t realize you were Chris from T3 :disguised_face:

Wasn’t gaslighting. I swear.

Email sent.

He’s not Chris from T3.

The guy on the T3 videos, who calls himself “Chris”, is @HoneyBadger from the TrueNAS forums.

Hi, a newbie question here. If I set up a 3 way mirror array special vdev, can I replace later on one by one the disks with others bigger and expand? Just want to make sure before making the move. Thanks!

I believe that’s the expected behavior. @NickF1227, can you confirm?

It should be, just like any other vdev.

@Constantin, I think you should add L2ARC tag to this topic.

That is a good idea. I think I will break it out as a separate resource because otherwise this likely becomes too long a post?

Is it still recommended to use Optane drive over using a PCIe 5.0 SSD? I was thinking something like the Samsung 9100 Pro.

For what use?

Also, you can’t just say Optane as it comes in different flavours. M10, 900/905p, P1600X etc which have different performance characteristics.

For a SLOG, the 900p/905p or 1600X (or better) are really good. Endurance, IOPS, Latency are all superb. Perfect for a SLOG. For anything else I would use the Samsung

Like @NugentS, I wonder why the focus on Optane for an sVDEV? The point of Optane is extreme write-tolerance, which makes it perfect for a SLOG that is written to relentlessly but (hopefully) rarely read. How data-center-centric your sVDEV drives should be re: wear tolerance is largely a question of use case.

sVDEVs with a lot of changing metadata / small file blocks (i.e. databases being housed entirely in the sVDEV, for example), should likely have a good deal of wear resistance. Pools that are more WORM likely are OK with much less wear-resistant SSDs. I chose datacenter SSDs from Intel for my pools sVDEV (despite it being largely WORM) because I like to sleep well at night. It’s the same reason I went Z3 when Z2 likely would have been good enough.

One thing I would look for in all your SSDs regardless of use case is whether they use a faster flash cache up front that is periodically flushed to slower flash in the rear. Those kinds of drives can really choke under sustained load, whether it’s sVDEV, SLOG, or even a SSD-pool for VMs or whatever. You really want to avoid these kinds of SSDs in a NAS.

It’s not just the uneven work performance but the potential for stuff to be still stuck in the volatile front-end cache and getting wiped out due to a problem (power loss, system halt, whatever) even though ZFS thought it was already written to the drive. That’s the problem with devices lying to ZFS about the CoW process being complete when it’s not and it’s also why proper SLOG devices will feature PLP to ensure the transactions on them can be written to the pool when the system starts up again.

I will try to address the small block issue tomorrow, it requires a re-write of the above. Then I will also try to address a separate resource page on L2ARC.

Endurance is one factor that makes them recommended, but Optane is also 1) extremely fast at the pure NAND level, meaning it skips over the issues with the pSLC and “front-loaded” performance you mention below, and 2) close to effectively immune to performance drops under mixed workloads vs. regular NAND.

Unfortunately Intel stopped making anything Optane in 2022/23.

https://www.servethehome.com/intel-optane-559m-impairment-with-q2-2022-wind-down/

SCM, which is a similar idea, is available from manufacturers like Kioxia

https://americas.kioxia.com/en-ca/business/ssd/enterprise-ssd/fl6.html

Allegedly Samsung is reviving Z-NAND with six products planned. FWIW.

Thank you so much for your guidance, as I’m setting up a new pool with svdev.

I created a pool of 4x 16TB HDDs in striped mirror configuration + 2x Intel Optane P1600X 118GB in Mirror config for metadata.

record size = 128k for most folders, except Movies & Music where the dataset is set to 1M

Small block size = 8k, based on my calculations according to the logic in the start post:

110GB usable space on Optanes → 25% = 27.5GB of metadata.

I just filled up the pool with the real data and I ran an analysis. The pool is filled at 45% (13TB used, 16TB available)

The svdev is filled with 9.94GB and 99.1GB available (9% filled)

I think that I chose the small block size too small - what do you think?

Based on the data: which small block size would you recommend instead? 16k, 32k, 64k?

If I read the data correctly, then:

16k = 9.61GB

32k = 18.8GB → 42GB at 100% (Optanes filled at 40%)

64k = 48GB → 106GB at 100% (Optanes filled at 96%)

Not happy, but now or never to adjust the small block size and transfer all data again.

Any thoughts about different small block sizes for different data sets?

truenas_admin@TNS-Fileserver[~]$ sudo zdb -bbbs Storage
[sudo] password for truenas_admin: 

Traversing all blocks to verify nothing leaked ...

loading concrete vdev 2, metaslab 108 of 109 ...
13.0T completed (249010MB/s) estimated time remaining: 0hr 00min 00sec        
        No leaks (block sum matches space maps exactly)

        bp count:              19692491
        ganged count:                 0
        bp logical:      14814583634944      avg: 752296
        bp physical:     14317959218688      avg: 727077     compression:   1.03
        bp allocated:    14323296903168      avg: 727348     compression:   1.03
        bp deduped:                   0    ref>1:      0   deduplication:   1.00
        bp cloned:                    0    count:      0
        Normal class:    14300706447360     used: 44.75%
        Special class       10669350912     used:  9.12%
        Embedded log class              0     used:  0.00%

        additional, non-pointer bps of type 0:         78
         number of (compressed) bytes:  number of bps
                         28:     17 *****************
                         29:     10 **********
                         30:      0 
                         31:      0 
                         32:      4 ****
                         33:     11 ***********
                         34:      0 
                         35:      0 
                         36:      0 
                         37:      0 
                         38:      0 
                         39:      0 
                         40:      2 **
                         41:      4 ****
                         42:     10 **********
                         43:      0 
                         44:      0 
                         45:      0 
                         46:      0 
                         47:      0 
                         48:      0 
                         49:      0 
                         50:      1 *
                         51:      0 
                         52:      0 
                         53:      0 
                         54:      0 
                         55:      0 
                         56:      0 
                         57:      0 
                         58:      0 
                         59:      0 
                         60:      0 
                         61:      0 
                         62:      0 
                         63:      0 
                         64:      2 **
                         65:      6 ******
                         66:      5 *****
                         67:      0 
                         68:      2 **
                         69:      0 
                         70:      0 
                         71:      0 
                         72:      0 
                         73:      0 
                         74:      2 **
                         75:      0 
                         76:      0 
                         77:      0 
                         78:      0 
                         79:      0 
                         80:      0 
                         81:      0 
                         82:      0 
                         83:      0 
                         84:      0 
                         85:      0 
                         86:      0 
                         87:      0 
                         88:      0 
                         89:      0 
                         90:      0 
                         91:      0 
                         92:      0 
                         93:      0 
                         94:      0 
                         95:      0 
                         96:      0 
                         97:      0 
                         98:      0 
                         99:      0 
                        100:      1 *
                        101:      0 
                        102:      1 *
        Dittoed blocks on same vdev: 942539

Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
     -      -       -       -       -       -        -  unallocated
     2    32K      8K     24K     12K    4.00     0.00  object directory
     2    64K      8K     24K     12K    8.00     0.00      L1 object array
    16     8K      8K    192K     12K    1.00     0.00      L0 object array
    18    72K     16K    216K     12K    4.50     0.00  object array
     1    16K      4K     12K     12K    4.00     0.00  packed nvlist
     -      -       -       -       -       -        -  packed nvlist size
     -      -       -       -       -       -        -  bpobj
     -      -       -       -       -       -        -  bpobj header
     -      -       -       -       -       -        -  SPA space map header
     8   128K     32K     96K     12K    4.00     0.00      L1 SPA space map
   977   122M   5.75M   17.3M   18.1K   21.22     0.00      L0 SPA space map
   985   122M   5.79M   17.4M   18.0K   21.13     0.00  SPA space map
     -      -       -       -       -       -        -  ZIL intent log
    15  1.88M     60K    120K      8K   32.00     0.00      L5 DMU dnode
    15  1.88M     60K    120K      8K   32.00     0.00      L4 DMU dnode
    15  1.88M     60K    120K      8K   32.00     0.00      L3 DMU dnode
    15  1.88M     60K    120K      8K   32.00     0.00      L2 DMU dnode
    44  5.50M   1.99M      4M   93.1K    2.76     0.00      L1 DMU dnode
 30.4K   486M    483M    967M   31.9K    1.01     0.01      L0 DMU dnode
 30.5K   499M    485M    972M   31.9K    1.03     0.01  DMU dnode
    16    64K     64K    132K   8.25K    1.00     0.00  DMU objset
     -      -       -       -       -       -        -  DSL directory
    18  10.5K      1K     12K     682   10.50     0.00  DSL directory child map
     -      -       -       -       -       -        -  DSL dataset snap map
    33   482K    120K    360K   10.9K    4.01     0.00  DSL props
     -      -       -       -       -       -        -  DSL dataset
     -      -       -       -       -       -        -  ZFS znode
     -      -       -       -       -       -        -  ZFS V0 ACL
     2    64K      8K     16K      8K    8.00     0.00      L3 ZFS plain file
 3.38K   108M   13.6M   27.2M   8.05K    7.95     0.00      L2 ZFS plain file
  587K  18.3G   2.68G   5.35G   9.34K    6.85     0.04      L1 ZFS plain file
 17.9M  13.4T   13.0T   13.0T    746K    1.03    99.93      L0 ZFS plain file
 18.5M  13.5T   13.0T   13.0T    723K    1.03    99.97  ZFS plain file
     1    32K      4K      8K      8K    8.00     0.00      L2 ZFS directory
 85.3K  2.67G    341M    683M   8.00K    8.00     0.00      L1 ZFS directory
  193K  2.74G    727M   1.54G   8.19K    3.86     0.01      L0 ZFS directory
  278K  5.40G   1.04G   2.21G   8.13K    5.18     0.02  ZFS directory
    15    15K     15K    120K      8K    1.00     0.00  ZFS master node
    15  7.50K   7.50K    120K      8K    1.00     0.00  ZFS delete queue
     -      -       -       -       -       -        -  zvol object
     -      -       -       -       -       -        -  zvol prop
     -      -       -       -       -       -        -  other uint8[]
     -      -       -       -       -       -        -  other uint64[]
     -      -       -       -       -       -        -  other ZAP
     -      -       -       -       -       -        -  persistent error log
     2   256K     20K     60K     30K   12.80     0.00  SPA history
     -      -       -       -       -       -        -  SPA history offsets
     -      -       -       -       -       -        -  Pool properties
     -      -       -       -       -       -        -  DSL permissions
     -      -       -       -       -       -        -  ZFS ACL
     -      -       -       -       -       -        -  ZFS SYSACL
     -      -       -       -       -       -        -  FUID table
     -      -       -       -       -       -        -  FUID table size
     1     1K      1K     12K     12K    1.00     0.00  DSL dataset next clones
     -      -       -       -       -       -        -  scan work queue
    45  22.5K   22.5K    360K      8K    1.00     0.00  ZFS user/group/project used
     -      -       -       -       -       -        -  ZFS user/group/project quota
     -      -       -       -       -       -        -  snapshot refcount tags
     -      -       -       -       -       -        -  DDT ZAP algorithm
     -      -       -       -       -       -        -  DDT statistics
 20.5K  12.8M   12.8M    164M   8.00K    1.00     0.00  System attributes
    15  7.50K   7.50K    120K      8K    1.00     0.00  SA master node
    15  22.5K   22.5K    120K      8K    1.00     0.00  SA attr registration
    30   480K    120K    240K      8K    4.00     0.00  SA attr layouts
     -      -       -       -       -       -        -  scan translations
     -      -       -       -       -       -        -  deduplicated block
     -      -       -       -       -       -        -  DSL deadlist map
     -      -       -       -       -       -        -  DSL deadlist map hdr
     1     1K      1K     12K     12K    1.00     0.00  DSL dir clones
     -      -       -       -       -       -        -  bpobj subobj
     -      -       -       -       -       -        -  deferred free
     -      -       -       -       -       -        -  dedup ditto
    81   960K    170K    696K   8.59K    5.63     0.00  other
    15  1.88M     60K    120K      8K   32.00     0.00      L5 Total
    15  1.88M     60K    120K      8K   32.00     0.00      L4 Total
    17  1.94M     68K    136K      8K   29.18     0.00      L3 Total
 3.40K   110M   13.7M   27.4M   8.05K    8.06     0.00      L2 Total
  672K  21.0G   3.01G   6.02G   9.18K    6.98     0.05      L1 Total
 18.1M  13.5T   13.0T   13.0T    736K    1.03    99.95      L0 Total
 18.8M  13.5T   13.0T   13.0T    710K    1.03   100.00  Total
  920K  24.5G   4.22G   8.72G   9.70K    5.79     0.07  Metadata Total

Block Size Histogram

  block   psize                lsize                asize
   size   Count   Size   Cum.  Count   Size   Cum.  Count   Size   Cum.
    512:  41.7K  20.9M  20.9M  41.7K  20.9M  20.9M      0      0      0
     1K:  19.5K  22.7M  43.6M  19.5K  22.7M  43.6M      0      0      0
     2K:  20.5K  54.9M  98.5M  20.5K  54.9M  98.5M      0      0      0
     4K:   895K  3.54G  3.63G  30.3K   172M   271M   108K   432M   432M
     8K:   196K  1.92G  5.55G  51.9K   590M   860M  1.01M  8.41G  8.83G
    16K:   206K  4.06G  9.61G   258K  4.37G  5.21G   171K  3.51G  12.3G
    32K:   214K  9.21G  18.8G   702K  22.3G  27.5G   278K  11.2G  23.5G
    64K:   327K  29.1G  48.0G  37.9K  3.49G  31.0G   328K  29.1G  52.7G
   128K:  4.14M   533G   581G  4.76M   610G   641G  4.14M   533G   586G
   256K:   128K  48.1G   629G  20.3K  7.76G   649G   128K  48.0G   634G
   512K:  1.04M   819G  1.41T  63.3K  37.1G   686G  1.04M   819G  1.42T
     1M:  11.6M  11.6T  13.0T  12.8M  12.8T  13.5T  11.6M  11.6T  13.0T
     2M:      0      0  13.0T      0      0  13.5T      0      0  13.0T
     4M:      0      0  13.0T      0      0  13.5T      0      0  13.0T
     8M:      0      0  13.0T      0      0  13.5T      0      0  13.0T
    16M:      0      0  13.0T      0      0  13.5T      0      0  13.0T

                            capacity   operations   bandwidth  ---- errors ----
description                used avail  read write  read write  read write cksum
Storage                   13.0T 16.2T 12.4K     0 63.1M     0     0     0     0
  mirror                  6.51T 8.03T     0     0 16.6K     0     0     0     0
    /dev/disk/by-partuuid/9e46fcc5-6b78-4d4f-b907-bcc1f12fee1d                              0     0 8.28K     0     0     0     0
    /dev/disk/by-partuuid/401c4d27-9570-40c9-81df-43a5f81e8c9d                              0     0 8.28K     0     0     0     0
  mirror                  6.49T 8.05T     0     0 16.6K     0     0     0     0
    /dev/disk/by-partuuid/3e2f7703-7f3e-46fd-9fbc-20f7e45d666c                              0     0 8.28K     0     0     0     0
    /dev/disk/by-partuuid/388c2c23-bb71-4dbf-9d6c-670de38a34e1                              0     0 8.28K     0     0     0     0
  mirror (special)        9.94G 99.1G 12.4K     0 63.1M     0     0     0     0
    /dev/disk/by-partuuid/b5ee8321-c573-45bf-950b-4b978e913b4f                          6.22K     0 31.6M     0     0     0     0
    /dev/disk/by-partuuid/fc947312-a4af-4d1f-b392-cf82067e2063                          6.19K     0 31.4M     0     0     0     0

Thank you for any input

I believe that the small blocks will only go to 75% (or summat like that - I am vaguely remembering - so may be wrong)

That would indicate to me that 32k would be right for the hardware

By default, small block storage is allocated 75% of the space on a sVDEV. It can be adjusted. Of that 75%, I would not fill it more than 80% because of ZFS. So on a default basis, planned small block storage should not exceed 60% of raw drive capacity (ie 0.75 x 0.8).

I wasn’t that far off.

:slight_smile:

What would be good would be to include something like this process (zdb -bbbs Storage), followed by calculations into the GUI when setting up a svdev.

Yeah, I need to do a better job at all that. Presently have a bunch of unrelated chores to finish so it’ll likely be the end of the weekend.