As zfs rewrite is mentioned briefly, when will the -P option be included in truenas?
in my case, it freed up quite a lot of space after an expansion of a raidz2, but apart from having to fiddle around with tmux, the main drawback seems to be the increase of snapshot size.
Yeah, well, L2ARC was likely added to ZFS more than 15 years ago, back when SSDs were in their infancy. Unlike some ZFS vDev types, L2ARC was designed to be easily removed. And if it fails, no data loss, just potentially reduced read speed.
I agree. Many people think L2ARC, (and possibly ARC), to behave as a Tiered Storage for ZFS. It sort of does, but the designers understood that some items read, are just read once. Like video files when streaming them. Thus, metadata would be more useful to store in ARC & L2ARC.
Even a simple copy out of a file, like a video file, does not make sense to cache it in ARC or L2ARC. (At least to ZFS.) The intent behind both is repetitive read accesses, over a shorter period of time.
You absolutely do NOT want ZFS scrubbing to flush ARC, (and potentially L2ARC). I am guessing you don’t mean ZFS scrubbing in the paragraph above. Did you mean video file seeking back and forth, (that I have heard some video editors call scrubbing)?
ARC is supposed to be smart, but in a general purpose way. Lots of corner cases exist that either need straight forward tuning using:
primarycache=all|none|metadata
secondarycache=all|none|metadata
Or the in depth tunables which can change function per ZFS version;
man/man4/zfs.4
Agreed, with the caveat that you have enough RAM for the pointers. Too little RAM and too large of a L2ARC device, ends up flushing ARC entries BEFORE ZFS thinks they should hang around longer, like in the L2ARC device.
I believe scrubbing in this context meant the action of scrolling through the video timeline in the editing programm, not a zfs scrub. While scrubbing a video timeline the file gets read repeatedly and therefore should not be evicted from arc/l2arc.
yes yes scrubbing like seeking 
(mvp right there) and yes a simple stream shouldn’t need to be cached but if the space is available it could cache them and keep the data around for future reading.
That’s one set of words I would never have thought to apply to a tape drive. 
Some early 26, some later 26. I’m being deliberately vague 
The framework for the special vdev feeds is there in OpenZFS 2.4.0 so we need to write some bits to easily enable it in middleware.
Re: L2ARC - as mentioned by @Arwen there’s population rules and ingest speed limits around L2ARC feeding so it doesn’t work exactly as stated. The special vdev is also beneficial for the metadata writes/updates - having your metadata in L2ARC doesn’t help that workflow, which can be a bottleneck at times.
It’s not just “boot delay” but “pool import delay” which is important in Enterprise cases with HA failover. 60 additional seconds during a failover is a Big Deal and usually enough to make most client workloads “respond poorly” aka fail/crash. The faster the system and L2ARC devices the shorter this window is, but it’s something we have to consider too.
oh i forgot about the fail over states.
*i’m coming here from level 7 administration (done a decade of google workspace implementations and consultations for big and small corporates as a founder of a gcloud partner). now i’m doing everything local on my own again
We’re thinking “TrueNAS 26” - again, deliberate vagueness here - should include some functionality to do a zfs rewrite on a per-dataset level, in order to make use of the ability to relocate data to/from special - note that -P won’t change the impact of the rewrite on snapshots, but it will mean a pool won’t flag all of its data for re-replication if snapshots don’t exist.
…but then wait for DNA-based storage to become mainstream and these tape drives will look positively blazing compared to a sequencer working through billions of base pairs.
Well, it is true.
Back in the day before Sun bought Storage Tek, I worked as an Unix SysAdmin at the Storage Tek headquarters. Naturally we used their own tape libraries, like the SL8500. It supported 2 different types of tape drives, one single reel for density. And one that had 2 spindles for faster seek times, (T9840), but at the cost of less tape / noticeably less density.
The following has details:
Can’t wait to upgrade to zfs on my body & have checksums performed before cellular replication. Am worried about how many additional cells I’d be expected to maintain for proper redundancy 
zbody set feature@braklul=enabled
Was hoping for something like this, but I’ll take what I can get I guess:
Different flavour of nerd, same genetic stability* (ignore chaos & rumours of unstable genomes in certain chapters).
I think in the future we’ll be more like tech priests… we will have to pray to the Machine gods for tech support…
Edit:
But for real, imagine you have an issue, call tech support and they tell you: sprinkle some holy oil on it and pray that it starts working again, and it does…
Clarke’s Third Law states so.
In the style of Zardoz? 
(Eternal respect to Sir Sean Connery for his performance in this costume!)
Meme: You’re tech support, tell user to reboot to solve his problem… and it works. 
Would a use case for metadata special device be to accelerate directory operations like ls -lR?
A less complex and safer way to do it is to work in this order:
zfs_arc_meta_balance parameter to favor metadata over datasecondarycache=metadataIncreasing RAM will help you with performance overall, not just with metadata. The ARC will have more room to work with.
A higher value for zfs_arc_meta_balance will decrease the likelihood of eviction of metadata from the ARC.
Finally, if for some reason metadata keeps being evicted aggressively, even with more RAM, a larger ARC, and a high value for zfs_arc_meta_balance, then you can add a fast NVMe as a persistent L2ARC device, so that metadata requests won’t need to be pulled from spinning HDDs. Unlike a special vdev, an L2ARC doesn’t need to be redundant and if it fails you won’t lose your entire pool.
Thank you – that’s probably a better general approach, but I’m also thinking of use cases where even the initial access is very slow. Directory operations involving the Immich thumbs directory for example (where it is designed with a structure like xx/xx/ resulting in 256 x 256 directories to look through) are painfully slow with spinning disk. find thumbs | wc -l goes on forever.
Maybe it is better to just put those kinds of use cases entirely on SSD (as the data size is relatively small, although I couldn’t really confirm because I’m not prepared to spend hours for du -sk thumbs to finish…) and not risk the main data pool.
In openzfs 2.4, special vdev will be used for ZIL in addition to metadata and small files. If one was going with a hybrid pool anyway, it sounds like in most cases one will be better off with a multi-purpose redundant special vdev than a single-purpose slog. Caveats are that a special vdev cannot be removed, loss of the special vdev will result in loss of the pool, and if added after pool creation existing metadata is not migrated. In my example of the Immich thumbs directory, I would need to zfs send/receive to force the metadata to move to the special vdev.
Wouldn’t that require special VDEVs to have all features such as power loss prevention, PLP, etc. How much data and time are you willing to lose? Reading current data is one thing but writing it to completion is, entirely, another.