Drop Kris and I a line - t3@truenas.com - and we’ll see if we can do a viewer-questions heavy show. There’s several caveats around sVDEV use (as this thread’s existence speaks to!) and using it safely.
Didn’t realize you were Chris from T3 ![]()
Wasn’t gaslighting. I swear.
Email sent.
He’s not Chris from T3.
The guy on the T3 videos, who calls himself “Chris”, is @HoneyBadger from the TrueNAS forums.
Hi, a newbie question here. If I set up a 3 way mirror array special vdev, can I replace later on one by one the disks with others bigger and expand? Just want to make sure before making the move. Thanks!
I believe that’s the expected behavior. @NickF1227, can you confirm?
It should be, just like any other vdev.
@Constantin, I think you should add L2ARC tag to this topic.
That is a good idea. I think I will break it out as a separate resource because otherwise this likely becomes too long a post?
Is it still recommended to use Optane drive over using a PCIe 5.0 SSD? I was thinking something like the Samsung 9100 Pro.
For what use?
Also, you can’t just say Optane as it comes in different flavours. M10, 900/905p, P1600X etc which have different performance characteristics.
For a SLOG, the 900p/905p or 1600X (or better) are really good. Endurance, IOPS, Latency are all superb. Perfect for a SLOG. For anything else I would use the Samsung
Like @NugentS, I wonder why the focus on Optane for an sVDEV? The point of Optane is extreme write-tolerance, which makes it perfect for a SLOG that is written to relentlessly but (hopefully) rarely read. How data-center-centric your sVDEV drives should be re: wear tolerance is largely a question of use case.
sVDEVs with a lot of changing metadata / small file blocks (i.e. databases being housed entirely in the sVDEV, for example), should likely have a good deal of wear resistance. Pools that are more WORM likely are OK with much less wear-resistant SSDs. I chose datacenter SSDs from Intel for my pools sVDEV (despite it being largely WORM) because I like to sleep well at night. It’s the same reason I went Z3 when Z2 likely would have been good enough.
One thing I would look for in all your SSDs regardless of use case is whether they use a faster flash cache up front that is periodically flushed to slower flash in the rear. Those kinds of drives can really choke under sustained load, whether it’s sVDEV, SLOG, or even a SSD-pool for VMs or whatever. You really want to avoid these kinds of SSDs in a NAS.
It’s not just the uneven work performance but the potential for stuff to be still stuck in the volatile front-end cache and getting wiped out due to a problem (power loss, system halt, whatever) even though ZFS thought it was already written to the drive. That’s the problem with devices lying to ZFS about the CoW process being complete when it’s not and it’s also why proper SLOG devices will feature PLP to ensure the transactions on them can be written to the pool when the system starts up again.
I will try to address the small block issue tomorrow, it requires a re-write of the above. Then I will also try to address a separate resource page on L2ARC.
Endurance is one factor that makes them recommended, but Optane is also 1) extremely fast at the pure NAND level, meaning it skips over the issues with the pSLC and “front-loaded” performance you mention below, and 2) close to effectively immune to performance drops under mixed workloads vs. regular NAND.
Unfortunately Intel stopped making anything Optane in 2022/23.
https://www.servethehome.com/intel-optane-559m-impairment-with-q2-2022-wind-down/
SCM, which is a similar idea, is available from manufacturers like Kioxia
https://americas.kioxia.com/en-ca/business/ssd/enterprise-ssd/fl6.html
Allegedly Samsung is reviving Z-NAND with six products planned. FWIW.
Thank you so much for your guidance, as I’m setting up a new pool with svdev.
I created a pool of 4x 16TB HDDs in striped mirror configuration + 2x Intel Optane P1600X 118GB in Mirror config for metadata.
record size = 128k for most folders, except Movies & Music where the dataset is set to 1M
Small block size = 8k, based on my calculations according to the logic in the start post:
110GB usable space on Optanes → 25% = 27.5GB of metadata.
I just filled up the pool with the real data and I ran an analysis. The pool is filled at 45% (13TB used, 16TB available)
The svdev is filled with 9.94GB and 99.1GB available (9% filled)
I think that I chose the small block size too small - what do you think?
Based on the data: which small block size would you recommend instead? 16k, 32k, 64k?
If I read the data correctly, then:
16k = 9.61GB
32k = 18.8GB → 42GB at 100% (Optanes filled at 40%)
64k = 48GB → 106GB at 100% (Optanes filled at 96%)
Not happy, but now or never to adjust the small block size and transfer all data again.
Any thoughts about different small block sizes for different data sets?
truenas_admin@TNS-Fileserver[~]$ sudo zdb -bbbs Storage
[sudo] password for truenas_admin:
Traversing all blocks to verify nothing leaked ...
loading concrete vdev 2, metaslab 108 of 109 ...
13.0T completed (249010MB/s) estimated time remaining: 0hr 00min 00sec
No leaks (block sum matches space maps exactly)
bp count: 19692491
ganged count: 0
bp logical: 14814583634944 avg: 752296
bp physical: 14317959218688 avg: 727077 compression: 1.03
bp allocated: 14323296903168 avg: 727348 compression: 1.03
bp deduped: 0 ref>1: 0 deduplication: 1.00
bp cloned: 0 count: 0
Normal class: 14300706447360 used: 44.75%
Special class 10669350912 used: 9.12%
Embedded log class 0 used: 0.00%
additional, non-pointer bps of type 0: 78
number of (compressed) bytes: number of bps
28: 17 *****************
29: 10 **********
30: 0
31: 0
32: 4 ****
33: 11 ***********
34: 0
35: 0
36: 0
37: 0
38: 0
39: 0
40: 2 **
41: 4 ****
42: 10 **********
43: 0
44: 0
45: 0
46: 0
47: 0
48: 0
49: 0
50: 1 *
51: 0
52: 0
53: 0
54: 0
55: 0
56: 0
57: 0
58: 0
59: 0
60: 0
61: 0
62: 0
63: 0
64: 2 **
65: 6 ******
66: 5 *****
67: 0
68: 2 **
69: 0
70: 0
71: 0
72: 0
73: 0
74: 2 **
75: 0
76: 0
77: 0
78: 0
79: 0
80: 0
81: 0
82: 0
83: 0
84: 0
85: 0
86: 0
87: 0
88: 0
89: 0
90: 0
91: 0
92: 0
93: 0
94: 0
95: 0
96: 0
97: 0
98: 0
99: 0
100: 1 *
101: 0
102: 1 *
Dittoed blocks on same vdev: 942539
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
- - - - - - - unallocated
2 32K 8K 24K 12K 4.00 0.00 object directory
2 64K 8K 24K 12K 8.00 0.00 L1 object array
16 8K 8K 192K 12K 1.00 0.00 L0 object array
18 72K 16K 216K 12K 4.50 0.00 object array
1 16K 4K 12K 12K 4.00 0.00 packed nvlist
- - - - - - - packed nvlist size
- - - - - - - bpobj
- - - - - - - bpobj header
- - - - - - - SPA space map header
8 128K 32K 96K 12K 4.00 0.00 L1 SPA space map
977 122M 5.75M 17.3M 18.1K 21.22 0.00 L0 SPA space map
985 122M 5.79M 17.4M 18.0K 21.13 0.00 SPA space map
- - - - - - - ZIL intent log
15 1.88M 60K 120K 8K 32.00 0.00 L5 DMU dnode
15 1.88M 60K 120K 8K 32.00 0.00 L4 DMU dnode
15 1.88M 60K 120K 8K 32.00 0.00 L3 DMU dnode
15 1.88M 60K 120K 8K 32.00 0.00 L2 DMU dnode
44 5.50M 1.99M 4M 93.1K 2.76 0.00 L1 DMU dnode
30.4K 486M 483M 967M 31.9K 1.01 0.01 L0 DMU dnode
30.5K 499M 485M 972M 31.9K 1.03 0.01 DMU dnode
16 64K 64K 132K 8.25K 1.00 0.00 DMU objset
- - - - - - - DSL directory
18 10.5K 1K 12K 682 10.50 0.00 DSL directory child map
- - - - - - - DSL dataset snap map
33 482K 120K 360K 10.9K 4.01 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS V0 ACL
2 64K 8K 16K 8K 8.00 0.00 L3 ZFS plain file
3.38K 108M 13.6M 27.2M 8.05K 7.95 0.00 L2 ZFS plain file
587K 18.3G 2.68G 5.35G 9.34K 6.85 0.04 L1 ZFS plain file
17.9M 13.4T 13.0T 13.0T 746K 1.03 99.93 L0 ZFS plain file
18.5M 13.5T 13.0T 13.0T 723K 1.03 99.97 ZFS plain file
1 32K 4K 8K 8K 8.00 0.00 L2 ZFS directory
85.3K 2.67G 341M 683M 8.00K 8.00 0.00 L1 ZFS directory
193K 2.74G 727M 1.54G 8.19K 3.86 0.01 L0 ZFS directory
278K 5.40G 1.04G 2.21G 8.13K 5.18 0.02 ZFS directory
15 15K 15K 120K 8K 1.00 0.00 ZFS master node
15 7.50K 7.50K 120K 8K 1.00 0.00 ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
- - - - - - - persistent error log
2 256K 20K 60K 30K 12.80 0.00 SPA history
- - - - - - - SPA history offsets
- - - - - - - Pool properties
- - - - - - - DSL permissions
- - - - - - - ZFS ACL
- - - - - - - ZFS SYSACL
- - - - - - - FUID table
- - - - - - - FUID table size
1 1K 1K 12K 12K 1.00 0.00 DSL dataset next clones
- - - - - - - scan work queue
45 22.5K 22.5K 360K 8K 1.00 0.00 ZFS user/group/project used
- - - - - - - ZFS user/group/project quota
- - - - - - - snapshot refcount tags
- - - - - - - DDT ZAP algorithm
- - - - - - - DDT statistics
20.5K 12.8M 12.8M 164M 8.00K 1.00 0.00 System attributes
15 7.50K 7.50K 120K 8K 1.00 0.00 SA master node
15 22.5K 22.5K 120K 8K 1.00 0.00 SA attr registration
30 480K 120K 240K 8K 4.00 0.00 SA attr layouts
- - - - - - - scan translations
- - - - - - - deduplicated block
- - - - - - - DSL deadlist map
- - - - - - - DSL deadlist map hdr
1 1K 1K 12K 12K 1.00 0.00 DSL dir clones
- - - - - - - bpobj subobj
- - - - - - - deferred free
- - - - - - - dedup ditto
81 960K 170K 696K 8.59K 5.63 0.00 other
15 1.88M 60K 120K 8K 32.00 0.00 L5 Total
15 1.88M 60K 120K 8K 32.00 0.00 L4 Total
17 1.94M 68K 136K 8K 29.18 0.00 L3 Total
3.40K 110M 13.7M 27.4M 8.05K 8.06 0.00 L2 Total
672K 21.0G 3.01G 6.02G 9.18K 6.98 0.05 L1 Total
18.1M 13.5T 13.0T 13.0T 736K 1.03 99.95 L0 Total
18.8M 13.5T 13.0T 13.0T 710K 1.03 100.00 Total
920K 24.5G 4.22G 8.72G 9.70K 5.79 0.07 Metadata Total
Block Size Histogram
block psize lsize asize
size Count Size Cum. Count Size Cum. Count Size Cum.
512: 41.7K 20.9M 20.9M 41.7K 20.9M 20.9M 0 0 0
1K: 19.5K 22.7M 43.6M 19.5K 22.7M 43.6M 0 0 0
2K: 20.5K 54.9M 98.5M 20.5K 54.9M 98.5M 0 0 0
4K: 895K 3.54G 3.63G 30.3K 172M 271M 108K 432M 432M
8K: 196K 1.92G 5.55G 51.9K 590M 860M 1.01M 8.41G 8.83G
16K: 206K 4.06G 9.61G 258K 4.37G 5.21G 171K 3.51G 12.3G
32K: 214K 9.21G 18.8G 702K 22.3G 27.5G 278K 11.2G 23.5G
64K: 327K 29.1G 48.0G 37.9K 3.49G 31.0G 328K 29.1G 52.7G
128K: 4.14M 533G 581G 4.76M 610G 641G 4.14M 533G 586G
256K: 128K 48.1G 629G 20.3K 7.76G 649G 128K 48.0G 634G
512K: 1.04M 819G 1.41T 63.3K 37.1G 686G 1.04M 819G 1.42T
1M: 11.6M 11.6T 13.0T 12.8M 12.8T 13.5T 11.6M 11.6T 13.0T
2M: 0 0 13.0T 0 0 13.5T 0 0 13.0T
4M: 0 0 13.0T 0 0 13.5T 0 0 13.0T
8M: 0 0 13.0T 0 0 13.5T 0 0 13.0T
16M: 0 0 13.0T 0 0 13.5T 0 0 13.0T
capacity operations bandwidth ---- errors ----
description used avail read write read write read write cksum
Storage 13.0T 16.2T 12.4K 0 63.1M 0 0 0 0
mirror 6.51T 8.03T 0 0 16.6K 0 0 0 0
/dev/disk/by-partuuid/9e46fcc5-6b78-4d4f-b907-bcc1f12fee1d 0 0 8.28K 0 0 0 0
/dev/disk/by-partuuid/401c4d27-9570-40c9-81df-43a5f81e8c9d 0 0 8.28K 0 0 0 0
mirror 6.49T 8.05T 0 0 16.6K 0 0 0 0
/dev/disk/by-partuuid/3e2f7703-7f3e-46fd-9fbc-20f7e45d666c 0 0 8.28K 0 0 0 0
/dev/disk/by-partuuid/388c2c23-bb71-4dbf-9d6c-670de38a34e1 0 0 8.28K 0 0 0 0
mirror (special) 9.94G 99.1G 12.4K 0 63.1M 0 0 0 0
/dev/disk/by-partuuid/b5ee8321-c573-45bf-950b-4b978e913b4f 6.22K 0 31.6M 0 0 0 0
/dev/disk/by-partuuid/fc947312-a4af-4d1f-b392-cf82067e2063 6.19K 0 31.4M 0 0 0 0
Thank you for any input
I believe that the small blocks will only go to 75% (or summat like that - I am vaguely remembering - so may be wrong)
That would indicate to me that 32k would be right for the hardware
By default, small block storage is allocated 75% of the space on a sVDEV. It can be adjusted. Of that 75%, I would not fill it more than 80% because of ZFS. So on a default basis, planned small block storage should not exceed 60% of raw drive capacity (ie 0.75 x 0.8).
I wasn’t that far off.
![]()
What would be good would be to include something like this process (zdb -bbbs Storage), followed by calculations into the GUI when setting up a svdev.
Yeah, I need to do a better job at all that. Presently have a bunch of unrelated chores to finish so it’ll likely be the end of the weekend.