Thank you for the input…
I was also thinking that 64k would be too close to the edge → 32k it is
That means to copy again 13TB ![]()
But then I will hopefully have a great setup for the next years.
Thank you for the input…
I was also thinking that 64k would be too close to the edge → 32k it is
That means to copy again 13TB ![]()
But then I will hopefully have a great setup for the next years.
Depends on your files.
I mean, does it hurt you in any way that the 37k size word file is read from the RAIDZ pool instead of the optanes?
Really comes down to what benefits from small block storage. I tailor my small block cutoff by dataset. For example, large image datasets are unlikely to benefit at all from small block storage on SSD (they benefit more from sVDEV speeding up metadata / directory traversal).
I set sVDEV small block size = recordsize for stuff I want to force into the sVDEV (i.e. Apps or Virtual Machines). Much of my other stuff is either archived (hence not compressible, not eligible for sVDEV small block transactions other than maybe some file end tails). So my sVDEV small block storage is pretty empty at the moment.
I would have a think about what parts of your data would benefit from sVDEV small block storage and make the settings there more aggressive (i.e. larger small block cutoff) while easing off small block storage for parts where it makes less of a difference. I.e. where are you reading and writing a lot vs. just stopping by occasionally?
For example, let’s say you are an avid Lightroom user and want to make the writing of the index / image metadata as fast as possible but cannot host your entire picture collection and image metadata on SSD. So, split image storage into a different dataset (recordsize = 1M, small block cutoff = 8k) while creating a second dataset just for the Lightroom index / image metadata where recordsize = 32kB = small block storage cutoff to force ALL the data that lightroom manipulates all the time into the sVDEV.
I dramatically reduced my needs for metadata and small block storage by putting a ton of content into archives that present themselves as 8GB files to TrueNAS. Combined with 1M recordsize and sVDEV, I get what I consider great performance without resorting to an all-flash pool.
I also think that the pool-wide special small blocks size setting doesn’t make sense. Moreover, it’s somewhat hard to come up with a workload/data type that would benefit from this. Your Lightroom metadata case is one of the few.
To get benefit, we need a workload that simultaneously reads a lot of files of mixed size. Not just traverse. The only trivial workload is… copying/transferring a directory with a lot of mixed-size files.
I’ll quibble a bit here. To me, it’s a matter whether the dataset is small enough to fit comfortably inside the small block storage part of the pool and whether I use said data a lot or at least enough.
For example, I have no doubt I would enjoy getting my iTunes library indices into the small block storage. Life would be zippy. But for itunes the current pool is fast enough and I use iTunes from the NAS sufficiently little that I cannot overcome my inner Schweinehund to get it done.
On the other hand, if and when I get Jellyfin running here, it’s metadata files, descriptions, etc. all will sit solely in the SSD portion of the pool and make scrolling among movies super quick and smooth. Meanwhile, the actual movies will sit solely in the HDD part of the pool, where they belong. ![]()
IMHO the better option for that example would be having Jellyfin (and its metatdata files and descriptions) on a SSD VM/Apps pool, while the movies themselves are on a HDD RAIDZ pool.
We can agree to disagree. The net difference in drive count is minimal but the impact on everyday pool performance of adding a sVDEV is significant. All things equal, I’d vote for a fusion pool over separate pools for that reason.
Either way, the sysadmin has to designate a different location for Jellyfin cache, transcoding scratch, and like storage that the default /mnt/.ix-apps/apps_catalog/jellyfin (?) location.
Finally got around to installing that stuff last night and realized what I really needed was emby, not jellyfin. Some of the misidentified titles are hilarious, will take some time to rectify.
I just finished the pool rebuilt with 32k.
sVDEV is now at 20GB (45% filled) - I will keep the 32k, as 64K is a big jump (if I read the block size diagram correctly).
Happy that I did it know, instead of “later”, which is usually never
truenas_admin@TNS-Fileserver[~]$ sudo zdb -bbbs Storage
[sudo] password for truenas_admin:
Traversing all blocks to verify nothing leaked ...
loading concrete vdev 2, metaslab 108 of 109 ...
12.9T completed (106383MB/s) estimated time remaining: 0hr 00min 01sec
No leaks (block sum matches space maps exactly)
bp count: 68811017
ganged count: 0
bp logical: 14697156133888 avg: 213587
bp physical: 14309994246656 avg: 207960 compression: 1.03
bp allocated: 14318614491136 avg: 208086 compression: 1.03
bp deduped: 0 ref>1: 0 deduplication: 1.00
bp cloned: 0 count: 0
Normal class: 14295588544512 used: 44.74%
Special class 21813592064 used: 18.64%
Embedded log class 0 used: 0.00%
additional, non-pointer bps of type 0: 4550
number of (compressed) bytes: number of bps
17: 15 *
18: 9 *
19: 4 *
20: 6 *
21: 5 *
22: 0
23: 6 *
24: 12 *
25: 4 *
26: 1 *
27: 4 *
28: 34 ***
29: 133 *********
30: 2 *
31: 3 *
32: 5 *
33: 35 ***
34: 4 *
35: 2 *
36: 1 *
37: 2 *
38: 4 *
39: 2 *
40: 2 *
41: 11 *
42: 28 **
43: 30 **
44: 13 *
45: 7 *
46: 6 *
47: 12 *
48: 8 *
49: 30 **
50: 63 *****
51: 25 **
52: 34 ***
53: 42 ***
54: 36 ***
55: 112 ********
56: 22 **
57: 68 *****
58: 589 **************************************
59: 49 ****
60: 13 *
61: 23 **
62: 19 **
63: 50 ****
64: 567 *************************************
65: 31 **
66: 29 **
67: 15 *
68: 14 *
69: 9 *
70: 13 *
71: 7 *
72: 14 *
73: 15 *
74: 16 **
75: 11 *
76: 9 *
77: 15 *
78: 19 **
79: 13 *
80: 20 **
81: 110 ********
82: 35 ***
83: 41 ***
84: 19 **
85: 18 **
86: 55 ****
87: 14 *
88: 13 *
89: 37 ***
90: 20 **
91: 16 **
92: 24 **
93: 5 *
94: 12 *
95: 11 *
96: 18 **
97: 11 *
98: 17 **
99: 12 *
100: 627 ****************************************
101: 8 *
102: 233 ***************
103: 13 *
104: 6 *
105: 14 *
106: 10 *
107: 13 *
108: 17 **
109: 65 *****
110: 517 *********************************
111: 109 *******
112: 28 **
Dittoed blocks on same vdev: 1108960
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
- - - - - - - unallocated
2 32K 8K 24K 12K 4.00 0.00 object directory
2 64K 8K 24K 12K 8.00 0.00 L1 object array
16 8K 8K 192K 12K 1.00 0.00 L0 object array
18 72K 16K 216K 12K 4.50 0.00 object array
1 16K 4K 12K 12K 4.00 0.00 packed nvlist
- - - - - - - packed nvlist size
- - - - - - - bpobj
- - - - - - - bpobj header
- - - - - - - SPA space map header
5 80K 20K 60K 12K 4.00 0.00 L1 SPA space map
969 121M 8.29M 24.9M 26.3K 14.61 0.00 L0 SPA space map
974 121M 8.31M 24.9M 26.2K 14.58 0.00 SPA space map
6 152K 152K 152K 25.3K 1.00 0.00 ZIL intent log
30 3.75M 120K 240K 8K 32.00 0.00 L5 DMU dnode
30 3.75M 120K 240K 8K 32.00 0.00 L4 DMU dnode
30 3.75M 120K 240K 8K 32.00 0.00 L3 DMU dnode
30 3.75M 120K 240K 8K 32.00 0.00 L2 DMU dnode
60 7.50M 2.21M 4.45M 75.9K 3.40 0.00 L1 DMU dnode
32.5K 519M 492M 986M 30.4K 1.06 0.01 L0 DMU dnode
32.6K 542M 495M 991M 30.4K 1.09 0.01 DMU dnode
31 124K 124K 252K 8.13K 1.00 0.00 DMU objset
- - - - - - - DSL directory
33 17K 2.50K 48K 1.45K 6.80 0.00 DSL directory child map
- - - - - - - DSL dataset snap map
59 838K 212K 672K 11.4K 3.95 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS V0 ACL
2 64K 8K 16K 8K 8.00 0.00 L3 ZFS plain file
105K 3.27G 419M 837M 8.00K 8.00 0.01 L2 ZFS plain file
642K 20.1G 5.15G 10.3G 16.4K 3.89 0.08 L1 ZFS plain file
64.6M 13.3T 13.0T 13.0T 206K 1.03 99.89 L0 ZFS plain file
65.3M 13.4T 13.0T 13.0T 204K 1.03 99.97 ZFS plain file
1 32K 4K 8K 8K 8.00 0.00 L2 ZFS directory
85.8K 2.68G 343M 687M 8.00K 8.00 0.01 L1 ZFS directory
199K 2.76G 734M 1.57G 8.07K 3.84 0.01 L0 ZFS directory
285K 5.44G 1.05G 2.24G 8.05K 5.17 0.02 ZFS directory
30 30K 30K 240K 8K 1.00 0.00 ZFS master node
30 15K 12.5K 200K 6.67K 1.20 0.00 ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
- - - - - - - persistent error log
2 256K 20K 60K 30K 12.80 0.00 SPA history
- - - - - - - SPA history offsets
- - - - - - - Pool properties
- - - - - - - DSL permissions
- - - - - - - ZFS ACL
- - - - - - - ZFS SYSACL
- - - - - - - FUID table
- - - - - - - FUID table size
1 2K 2K 12K 12K 1.00 0.00 DSL dataset next clones
- - - - - - - scan work queue
90 46K 39.5K 616K 6.84K 1.16 0.00 ZFS user/group/project used
- - - - - - - ZFS user/group/project quota
- - - - - - - snapshot refcount tags
- - - - - - - DDT ZAP algorithm
- - - - - - - DDT statistics
20.5K 12.8M 12.8M 164M 8.00K 1.00 0.00 System attributes
30 15K 12.5K 200K 6.67K 1.20 0.00 SA master node
30 45K 45K 240K 8K 1.00 0.00 SA attr registration
60 960K 240K 480K 8K 4.00 0.00 SA attr layouts
- - - - - - - scan translations
- - - - - - - deduplicated block
- - - - - - - DSL deadlist map
- - - - - - - DSL deadlist map hdr
1 2K 2K 12K 12K 1.00 0.00 DSL dir clones
- - - - - - - bpobj subobj
- - - - - - - deferred free
- - - - - - - dedup ditto
126 1.26M 256K 1.07M 8.67K 5.04 0.00 other
30 3.75M 120K 240K 8K 32.00 0.00 L5 Total
30 3.75M 120K 240K 8K 32.00 0.00 L4 Total
32 3.81M 128K 256K 8K 30.50 0.00 L3 Total
105K 3.27G 419M 837M 8.00K 8.01 0.01 L2 Total
728K 22.8G 5.49G 11.0G 15.4K 4.14 0.08 L1 Total
64.8M 13.3T 13.0T 13.0T 206K 1.03 99.91 L0 Total
65.6M 13.4T 13.0T 13.0T 203K 1.03 100.00 Total
1.06M 29.4G 7.12G 14.5G 13.7K 4.13 0.11 Metadata Total
Block Size Histogram
block psize lsize asize
size Count Size Cum. Count Size Cum. Count Size Cum.
512: 45.4K 22.7M 22.7M 45.4K 22.7M 22.7M 0 0 0
1K: 30.1K 35.8M 58.5M 30.1K 35.8M 58.5M 0 0 0
2K: 38.3K 103M 162M 38.3K 103M 162M 0 0 0
4K: 833K 3.29G 3.45G 39.7K 221M 383M 183K 731M 731M
8K: 258K 2.51G 5.96G 56.4K 636M 1018M 847K 6.82G 7.54G
16K: 315K 5.73G 11.7G 263K 4.48G 5.47G 300K 5.95G 13.5G
32K: 217K 9.65G 21.3G 860K 27.2G 32.7G 406K 15.5G 29.0G
64K: 2.16M 213G 235G 38.1K 3.51G 36.2G 2.16M 213G 242G
128K: 55.7M 6.97T 7.20T 58.2M 7.27T 7.31T 55.7M 6.97T 7.21T
256K: 85.7K 32.0G 7.23T 40.5K 15.0G 7.33T 85.9K 32.0G 7.24T
512K: 783K 616G 7.83T 36.9K 26.0G 7.35T 784K 616G 7.84T
1M: 5.18M 5.18T 13.0T 6.02M 6.02T 13.4T 5.18M 5.18T 13.0T
2M: 0 0 13.0T 0 0 13.4T 0 0 13.0T
4M: 0 0 13.0T 0 0 13.4T 0 0 13.0T
8M: 0 0 13.0T 0 0 13.4T 0 0 13.0T
16M: 0 0 13.0T 0 0 13.4T 0 0 13.0T
capacity operations bandwidth ---- errors ----
description used avail read write read write read write cksum
Storage 13.0T 16.2T 6.61K 0 50.0M 0 0 0 0
mirror 6.50T 8.05T 0 0 7.21K 0 0 0 0
/dev/disk/by-partuuid/50c45f2e-9182-4c59-b6c9-f1d06047e2ad 0 0 3.60K 0 0 0 0
/dev/disk/by-partuuid/35f2dfe7-8c17-4424-ad9a-98b7409c9da2 0 0 3.60K 0 0 0 0
mirror 6.50T 8.05T 0 0 7.21K 0 0 0 0
/dev/disk/by-partuuid/f3f90484-b1e5-465a-9e78-db27eb579fb6 0 0 3.60K 0 0 0 0
/dev/disk/by-partuuid/cb849f73-862d-4e2a-8534-9ab080bed295 0 0 3.60K 0 0 0 0
mirror (special) 20.3G 88.7G 6.61K 0 50.0M 0 0 0 0
/dev/disk/by-partuuid/dbcb5c5a-f6a3-43d5-a4e5-8be4ad1f6aa6 3.30K 0 24.9M 0 0 0 0
/dev/disk/by-partuuid/d98c515b-c29c-4cd6-b07a-58b23a1a5de5 3.31K 0 25.1M 0 0 0 0
i have a pool i am looking into adding a special vdev to it is 30TB in size around 7tb used.
I have a pair of optane h20 1tb nvme drive i was looking at for using as a special vdev in a mirror.
I see we can also use this to cache small files, like thumbnails and such to speed up seeks. Below is the output from my pool and the file sizes. I was wondering if i could get some direction as in choosing a small file size to use with this?
find . -type f -print0 | xargs -0 ls -l | awk ‘{ n=int(log($5)/log(2)); if (n<10) { n=10; } size[n]++ } END { for (i in size) printf(“%d %d\n”, 2^i, size[i]) }’ | sort -n | awk ‘function human(x) { x[1]/=1024; if (x[1]>=1024) { x[2]++; human(x) } } { a[1]=$1; a[2]=0; human(a); printf(“%3d%s: %6d\n”, a[1],substr(“kMGTEPYZ”,a[2]+1,1),$2) }’
1k: 94333
2k: 15741
4k: 11650
8k: 22795
16k: 51919
32k: 48252
64k: 32258
128k: 18241
256k: 15469
512k: 10121
1M: 7938
2M: 5456
4M: 4208
8M: 3664
16M: 3166
32M: 3515
64M: 2550
128M: 1242
256M: 758
512M: 424
1G: 180
2G: 198
4G: 68
8G: 25
16G: 15
32G: 3
64G: 1
128G: 2
256G: 1
IMHO thumbs and seeks are two completely different things.
Thumbs I doubt that you gain much, since most likely even reading them from an HDD is fast enough. Seek on the other hand would be metadata. That would profit extremely from a svdev.
So use the default of 0 (only metadata) is probably pretty good for a Jellyfin(?) use case.
Bad choice for svdev: These are 1 TB QLC drives with a little Optane bolted on, and QLC suffers with small transactions.
Depending on your platform it is even possible that these H20 do not work as intended: Intel has dropped support for the required x2/x2 bifurcation.
i do have a bifurcation card, would these do better as a cache vdev then?
Jellyfin/plex and backups /network shares
Definitely 0 then.
Don’t make my mistake and try to cache small files like pictures and then end up with tails from movies cached ![]()
You can specify on the basis of per dataset. ![]()
Set a very low small block value for movie datasets, set a higher value for databases and like content.
Just like recordsizes, adjusting small block on a dataset / use case basis is a good idea.
That is based on the conclusion that movie posters will load faster because of svdev. But counter intuitively, you might not really see big performance boost from that, since you might
so I would probably still leave it at 0 and not make my mistake and set it to 128k ![]()
Or IMHO even better, go do the “separate application from data” approach.
Jellyfin with its thumbnails and DB and OS is on a SSD mirror and accessing the movies from a dataset with a big recordsize.
“This is the way. “
Though I would argue it’s trivial to put the entire dataset for all apps in the sVDEV. In my case, it’s emby, among other apps, that resides in a dataset that is forced into the sVDEV by virtue of making its small block cutoff = the recordsize.
That is my setup. Jellyfin lives in proxmox, with all caches being local. The truenas media share is mounted on proxmox itself. Navigating in the UI is very fast (at least in LAN), and there is some minor delay when starting playback.
Any thoughts on using zfs rewrite to do the rebalancing? I added a 2x 2TB SSD mirror as a special vdev to my 6x 18TB HDD raidz2 vdev in my big pool; just wondering whether to use the zfs-inplace-rebalancing script above or zfs rewrite.
You should 100% use the zfs rewrite functionality here. The inplace-rebalancing requires exclusive/locked access to the files - zfs rewrite doesn’t.
Mind you, both will increase data footprint if snapshots are in play (which they should be)