Special VDEV (sVDEV) Planning, Sizing, and Considerations

Thank you for the input…

I was also thinking that 64k would be too close to the edge → 32k it is

That means to copy again 13TB :joy:

But then I will hopefully have a great setup for the next years.

Depends on your files.

I mean, does it hurt you in any way that the 37k size word file is read from the RAIDZ pool instead of the optanes?

Really comes down to what benefits from small block storage. I tailor my small block cutoff by dataset. For example, large image datasets are unlikely to benefit at all from small block storage on SSD (they benefit more from sVDEV speeding up metadata / directory traversal).

I set sVDEV small block size = recordsize for stuff I want to force into the sVDEV (i.e. Apps or Virtual Machines). Much of my other stuff is either archived (hence not compressible, not eligible for sVDEV small block transactions other than maybe some file end tails). So my sVDEV small block storage is pretty empty at the moment.

I would have a think about what parts of your data would benefit from sVDEV small block storage and make the settings there more aggressive (i.e. larger small block cutoff) while easing off small block storage for parts where it makes less of a difference. I.e. where are you reading and writing a lot vs. just stopping by occasionally?

For example, let’s say you are an avid Lightroom user and want to make the writing of the index / image metadata as fast as possible but cannot host your entire picture collection and image metadata on SSD. So, split image storage into a different dataset (recordsize = 1M, small block cutoff = 8k) while creating a second dataset just for the Lightroom index / image metadata where recordsize = 32kB = small block storage cutoff to force ALL the data that lightroom manipulates all the time into the sVDEV.

I dramatically reduced my needs for metadata and small block storage by putting a ton of content into archives that present themselves as 8GB files to TrueNAS. Combined with 1M recordsize and sVDEV, I get what I consider great performance without resorting to an all-flash pool.

I also think that the pool-wide special small blocks size setting doesn’t make sense. Moreover, it’s somewhat hard to come up with a workload/data type that would benefit from this. Your Lightroom metadata case is one of the few.

To get benefit, we need a workload that simultaneously reads a lot of files of mixed size. Not just traverse. The only trivial workload is… copying/transferring a directory with a lot of mixed-size files.

I’ll quibble a bit here. To me, it’s a matter whether the dataset is small enough to fit comfortably inside the small block storage part of the pool and whether I use said data a lot or at least enough.

For example, I have no doubt I would enjoy getting my iTunes library indices into the small block storage. Life would be zippy. But for itunes the current pool is fast enough and I use iTunes from the NAS sufficiently little that I cannot overcome my inner Schweinehund to get it done.

On the other hand, if and when I get Jellyfin running here, it’s metadata files, descriptions, etc. all will sit solely in the SSD portion of the pool and make scrolling among movies super quick and smooth. Meanwhile, the actual movies will sit solely in the HDD part of the pool, where they belong. :grinning_face_with_smiling_eyes:

IMHO the better option for that example would be having Jellyfin (and its metatdata files and descriptions) on a SSD VM/Apps pool, while the movies themselves are on a HDD RAIDZ pool.

We can agree to disagree. The net difference in drive count is minimal but the impact on everyday pool performance of adding a sVDEV is significant. All things equal, I’d vote for a fusion pool over separate pools for that reason.

Either way, the sysadmin has to designate a different location for Jellyfin cache, transcoding scratch, and like storage that the default /mnt/.ix-apps/apps_catalog/jellyfin (?) location.

Finally got around to installing that stuff last night and realized what I really needed was emby, not jellyfin. Some of the misidentified titles are hilarious, will take some time to rectify.

I just finished the pool rebuilt with 32k.

sVDEV is now at 20GB (45% filled) - I will keep the 32k, as 64K is a big jump (if I read the block size diagram correctly).

Happy that I did it know, instead of “later”, which is usually never

truenas_admin@TNS-Fileserver[~]$ sudo zdb -bbbs Storage
[sudo] password for truenas_admin: 

Traversing all blocks to verify nothing leaked ...

loading concrete vdev 2, metaslab 108 of 109 ...
12.9T completed (106383MB/s) estimated time remaining: 0hr 00min 01sec        
        No leaks (block sum matches space maps exactly)

        bp count:              68811017
        ganged count:                 0
        bp logical:      14697156133888      avg: 213587
        bp physical:     14309994246656      avg: 207960     compression:   1.03
        bp allocated:    14318614491136      avg: 208086     compression:   1.03
        bp deduped:                   0    ref>1:      0   deduplication:   1.00
        bp cloned:                    0    count:      0
        Normal class:    14295588544512     used: 44.74%
        Special class       21813592064     used: 18.64%
        Embedded log class              0     used:  0.00%

        additional, non-pointer bps of type 0:       4550
         number of (compressed) bytes:  number of bps
                         17:     15 *
                         18:      9 *
                         19:      4 *
                         20:      6 *
                         21:      5 *
                         22:      0 
                         23:      6 *
                         24:     12 *
                         25:      4 *
                         26:      1 *
                         27:      4 *
                         28:     34 ***
                         29:    133 *********
                         30:      2 *
                         31:      3 *
                         32:      5 *
                         33:     35 ***
                         34:      4 *
                         35:      2 *
                         36:      1 *
                         37:      2 *
                         38:      4 *
                         39:      2 *
                         40:      2 *
                         41:     11 *
                         42:     28 **
                         43:     30 **
                         44:     13 *
                         45:      7 *
                         46:      6 *
                         47:     12 *
                         48:      8 *
                         49:     30 **
                         50:     63 *****
                         51:     25 **
                         52:     34 ***
                         53:     42 ***
                         54:     36 ***
                         55:    112 ********
                         56:     22 **
                         57:     68 *****
                         58:    589 **************************************
                         59:     49 ****
                         60:     13 *
                         61:     23 **
                         62:     19 **
                         63:     50 ****
                         64:    567 *************************************
                         65:     31 **
                         66:     29 **
                         67:     15 *
                         68:     14 *
                         69:      9 *
                         70:     13 *
                         71:      7 *
                         72:     14 *
                         73:     15 *
                         74:     16 **
                         75:     11 *
                         76:      9 *
                         77:     15 *
                         78:     19 **
                         79:     13 *
                         80:     20 **
                         81:    110 ********
                         82:     35 ***
                         83:     41 ***
                         84:     19 **
                         85:     18 **
                         86:     55 ****
                         87:     14 *
                         88:     13 *
                         89:     37 ***
                         90:     20 **
                         91:     16 **
                         92:     24 **
                         93:      5 *
                         94:     12 *
                         95:     11 *
                         96:     18 **
                         97:     11 *
                         98:     17 **
                         99:     12 *
                        100:    627 ****************************************
                        101:      8 *
                        102:    233 ***************
                        103:     13 *
                        104:      6 *
                        105:     14 *
                        106:     10 *
                        107:     13 *
                        108:     17 **
                        109:     65 *****
                        110:    517 *********************************
                        111:    109 *******
                        112:     28 **
        Dittoed blocks on same vdev: 1108960

Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
     -      -       -       -       -       -        -  unallocated
     2    32K      8K     24K     12K    4.00     0.00  object directory
     2    64K      8K     24K     12K    8.00     0.00      L1 object array
    16     8K      8K    192K     12K    1.00     0.00      L0 object array
    18    72K     16K    216K     12K    4.50     0.00  object array
     1    16K      4K     12K     12K    4.00     0.00  packed nvlist
     -      -       -       -       -       -        -  packed nvlist size
     -      -       -       -       -       -        -  bpobj
     -      -       -       -       -       -        -  bpobj header
     -      -       -       -       -       -        -  SPA space map header
     5    80K     20K     60K     12K    4.00     0.00      L1 SPA space map
   969   121M   8.29M   24.9M   26.3K   14.61     0.00      L0 SPA space map
   974   121M   8.31M   24.9M   26.2K   14.58     0.00  SPA space map
     6   152K    152K    152K   25.3K    1.00     0.00  ZIL intent log
    30  3.75M    120K    240K      8K   32.00     0.00      L5 DMU dnode
    30  3.75M    120K    240K      8K   32.00     0.00      L4 DMU dnode
    30  3.75M    120K    240K      8K   32.00     0.00      L3 DMU dnode
    30  3.75M    120K    240K      8K   32.00     0.00      L2 DMU dnode
    60  7.50M   2.21M   4.45M   75.9K    3.40     0.00      L1 DMU dnode
 32.5K   519M    492M    986M   30.4K    1.06     0.01      L0 DMU dnode
 32.6K   542M    495M    991M   30.4K    1.09     0.01  DMU dnode
    31   124K    124K    252K   8.13K    1.00     0.00  DMU objset
     -      -       -       -       -       -        -  DSL directory
    33    17K   2.50K     48K   1.45K    6.80     0.00  DSL directory child map
     -      -       -       -       -       -        -  DSL dataset snap map
    59   838K    212K    672K   11.4K    3.95     0.00  DSL props
     -      -       -       -       -       -        -  DSL dataset
     -      -       -       -       -       -        -  ZFS znode
     -      -       -       -       -       -        -  ZFS V0 ACL
     2    64K      8K     16K      8K    8.00     0.00      L3 ZFS plain file
  105K  3.27G    419M    837M   8.00K    8.00     0.01      L2 ZFS plain file
  642K  20.1G   5.15G   10.3G   16.4K    3.89     0.08      L1 ZFS plain file
 64.6M  13.3T   13.0T   13.0T    206K    1.03    99.89      L0 ZFS plain file
 65.3M  13.4T   13.0T   13.0T    204K    1.03    99.97  ZFS plain file
     1    32K      4K      8K      8K    8.00     0.00      L2 ZFS directory
 85.8K  2.68G    343M    687M   8.00K    8.00     0.01      L1 ZFS directory
  199K  2.76G    734M   1.57G   8.07K    3.84     0.01      L0 ZFS directory
  285K  5.44G   1.05G   2.24G   8.05K    5.17     0.02  ZFS directory
    30    30K     30K    240K      8K    1.00     0.00  ZFS master node
    30    15K   12.5K    200K   6.67K    1.20     0.00  ZFS delete queue
     -      -       -       -       -       -        -  zvol object
     -      -       -       -       -       -        -  zvol prop
     -      -       -       -       -       -        -  other uint8[]
     -      -       -       -       -       -        -  other uint64[]
     -      -       -       -       -       -        -  other ZAP
     -      -       -       -       -       -        -  persistent error log
     2   256K     20K     60K     30K   12.80     0.00  SPA history
     -      -       -       -       -       -        -  SPA history offsets
     -      -       -       -       -       -        -  Pool properties
     -      -       -       -       -       -        -  DSL permissions
     -      -       -       -       -       -        -  ZFS ACL
     -      -       -       -       -       -        -  ZFS SYSACL
     -      -       -       -       -       -        -  FUID table
     -      -       -       -       -       -        -  FUID table size
     1     2K      2K     12K     12K    1.00     0.00  DSL dataset next clones
     -      -       -       -       -       -        -  scan work queue
    90    46K   39.5K    616K   6.84K    1.16     0.00  ZFS user/group/project used
     -      -       -       -       -       -        -  ZFS user/group/project quota
     -      -       -       -       -       -        -  snapshot refcount tags
     -      -       -       -       -       -        -  DDT ZAP algorithm
     -      -       -       -       -       -        -  DDT statistics
 20.5K  12.8M   12.8M    164M   8.00K    1.00     0.00  System attributes
    30    15K   12.5K    200K   6.67K    1.20     0.00  SA master node
    30    45K     45K    240K      8K    1.00     0.00  SA attr registration
    60   960K    240K    480K      8K    4.00     0.00  SA attr layouts
     -      -       -       -       -       -        -  scan translations
     -      -       -       -       -       -        -  deduplicated block
     -      -       -       -       -       -        -  DSL deadlist map
     -      -       -       -       -       -        -  DSL deadlist map hdr
     1     2K      2K     12K     12K    1.00     0.00  DSL dir clones
     -      -       -       -       -       -        -  bpobj subobj
     -      -       -       -       -       -        -  deferred free
     -      -       -       -       -       -        -  dedup ditto
   126  1.26M    256K   1.07M   8.67K    5.04     0.00  other
    30  3.75M    120K    240K      8K   32.00     0.00      L5 Total
    30  3.75M    120K    240K      8K   32.00     0.00      L4 Total
    32  3.81M    128K    256K      8K   30.50     0.00      L3 Total
  105K  3.27G    419M    837M   8.00K    8.01     0.01      L2 Total
  728K  22.8G   5.49G   11.0G   15.4K    4.14     0.08      L1 Total
 64.8M  13.3T   13.0T   13.0T    206K    1.03    99.91      L0 Total
 65.6M  13.4T   13.0T   13.0T    203K    1.03   100.00  Total
 1.06M  29.4G   7.12G   14.5G   13.7K    4.13     0.11  Metadata Total

Block Size Histogram

  block   psize                lsize                asize
   size   Count   Size   Cum.  Count   Size   Cum.  Count   Size   Cum.
    512:  45.4K  22.7M  22.7M  45.4K  22.7M  22.7M      0      0      0
     1K:  30.1K  35.8M  58.5M  30.1K  35.8M  58.5M      0      0      0
     2K:  38.3K   103M   162M  38.3K   103M   162M      0      0      0
     4K:   833K  3.29G  3.45G  39.7K   221M   383M   183K   731M   731M
     8K:   258K  2.51G  5.96G  56.4K   636M  1018M   847K  6.82G  7.54G
    16K:   315K  5.73G  11.7G   263K  4.48G  5.47G   300K  5.95G  13.5G
    32K:   217K  9.65G  21.3G   860K  27.2G  32.7G   406K  15.5G  29.0G
    64K:  2.16M   213G   235G  38.1K  3.51G  36.2G  2.16M   213G   242G
   128K:  55.7M  6.97T  7.20T  58.2M  7.27T  7.31T  55.7M  6.97T  7.21T
   256K:  85.7K  32.0G  7.23T  40.5K  15.0G  7.33T  85.9K  32.0G  7.24T
   512K:   783K   616G  7.83T  36.9K  26.0G  7.35T   784K   616G  7.84T
     1M:  5.18M  5.18T  13.0T  6.02M  6.02T  13.4T  5.18M  5.18T  13.0T
     2M:      0      0  13.0T      0      0  13.4T      0      0  13.0T
     4M:      0      0  13.0T      0      0  13.4T      0      0  13.0T
     8M:      0      0  13.0T      0      0  13.4T      0      0  13.0T
    16M:      0      0  13.0T      0      0  13.4T      0      0  13.0T

                            capacity   operations   bandwidth  ---- errors ----
description                used avail  read write  read write  read write cksum
Storage                   13.0T 16.2T 6.61K     0 50.0M     0     0     0     0
  mirror                  6.50T 8.05T     0     0 7.21K     0     0     0     0
    /dev/disk/by-partuuid/50c45f2e-9182-4c59-b6c9-f1d06047e2ad                              0     0 3.60K     0     0     0     0
    /dev/disk/by-partuuid/35f2dfe7-8c17-4424-ad9a-98b7409c9da2                              0     0 3.60K     0     0     0     0
  mirror                  6.50T 8.05T     0     0 7.21K     0     0     0     0
    /dev/disk/by-partuuid/f3f90484-b1e5-465a-9e78-db27eb579fb6                              0     0 3.60K     0     0     0     0
    /dev/disk/by-partuuid/cb849f73-862d-4e2a-8534-9ab080bed295                              0     0 3.60K     0     0     0     0
  mirror (special)        20.3G 88.7G 6.61K     0 50.0M     0     0     0     0
    /dev/disk/by-partuuid/dbcb5c5a-f6a3-43d5-a4e5-8be4ad1f6aa6                          3.30K     0 24.9M     0     0     0     0
    /dev/disk/by-partuuid/d98c515b-c29c-4cd6-b07a-58b23a1a5de5                          3.31K     0 25.1M     0     0     0     0

i have a pool i am looking into adding a special vdev to it is 30TB in size around 7tb used.

I have a pair of optane h20 1tb nvme drive i was looking at for using as a special vdev in a mirror.

I see we can also use this to cache small files, like thumbnails and such to speed up seeks. Below is the output from my pool and the file sizes. I was wondering if i could get some direction as in choosing a small file size to use with this?

find . -type f -print0 | xargs -0 ls -l | awk ‘{ n=int(log($5)/log(2)); if (n<10) { n=10; } size[n]++ } END { for (i in size) printf(“%d %d\n”, 2^i, size[i]) }’ | sort -n | awk ‘function human(x) { x[1]/=1024; if (x[1]>=1024) { x[2]++; human(x) } } { a[1]=$1; a[2]=0; human(a); printf(“%3d%s: %6d\n”, a[1],substr(“kMGTEPYZ”,a[2]+1,1),$2) }’

1k: 94333
2k: 15741
4k: 11650
8k: 22795
16k: 51919
32k: 48252
64k: 32258
128k: 18241
256k: 15469
512k: 10121
1M: 7938
2M: 5456
4M: 4208
8M: 3664
16M: 3166
32M: 3515
64M: 2550
128M: 1242
256M: 758
512M: 424
1G: 180
2G: 198
4G: 68
8G: 25
16G: 15
32G: 3
64G: 1
128G: 2
256G: 1

IMHO thumbs and seeks are two completely different things.

Thumbs I doubt that you gain much, since most likely even reading them from an HDD is fast enough. Seek on the other hand would be metadata. That would profit extremely from a svdev.

So use the default of 0 (only metadata) is probably pretty good for a Jellyfin(?) use case.

Bad choice for svdev: These are 1 TB QLC drives with a little Optane bolted on, and QLC suffers with small transactions.
Depending on your platform it is even possible that these H20 do not work as intended: Intel has dropped support for the required x2/x2 bifurcation.

i do have a bifurcation card, would these do better as a cache vdev then?

Jellyfin/plex and backups /network shares

Definitely 0 then.

Don’t make my mistake and try to cache small files like pictures and then end up with tails from movies cached :wink:

You can specify on the basis of per dataset. :grinning_face:

Set a very low small block value for movie datasets, set a higher value for databases and like content.

Just like recordsizes, adjusting small block on a dataset / use case basis is a good idea.

That is based on the conclusion that movie posters will load faster because of svdev. But counter intuitively, you might not really see big performance boost from that, since you might

  • have other bottlenecks
  • have cached that in RAM already
  • have cached that in webserver already
  • have cached that in proxy already
  • have cached that in ARC

so I would probably still leave it at 0 and not make my mistake and set it to 128k :slight_smile:

Or IMHO even better, go do the “separate application from data” approach.
Jellyfin with its thumbnails and DB and OS is on a SSD mirror and accessing the movies from a dataset with a big recordsize.

“This is the way. “

Though I would argue it’s trivial to put the entire dataset for all apps in the sVDEV. In my case, it’s emby, among other apps, that resides in a dataset that is forced into the sVDEV by virtue of making its small block cutoff = the recordsize.

That is my setup. Jellyfin lives in proxmox, with all caches being local. The truenas media share is mounted on proxmox itself. Navigating in the UI is very fast (at least in LAN), and there is some minor delay when starting playback.

Any thoughts on using zfs rewrite to do the rebalancing? I added a 2x 2TB SSD mirror as a special vdev to my 6x 18TB HDD raidz2 vdev in my big pool; just wondering whether to use the zfs-inplace-rebalancing script above or zfs rewrite.

You should 100% use the zfs rewrite functionality here. The inplace-rebalancing requires exclusive/locked access to the files - zfs rewrite doesn’t.

Mind you, both will increase data footprint if snapshots are in play (which they should be)