One thing all HDD share is that the first third of the disk is much faster than the last third. This drives me a bit nuts because on my primary RAIDZ-1 volume I run it at about 65% full with that data being mostly inactive. All my heavy usage in on new files that I add. This means that all my inactive data is on the fastest portion of the disks, and all my active files are on the slowest portion of the disks. This has me asking if there is a way to reverse this state. Would it be possible to … empty the pool completely, fill the first 35% of space with garbage, then add my 65% mostly static content. Then delete the 35% garbage? Would something like this not leave me with the static content on the slow portions of the disk, and all my new activity would occur on the fastest portions? Is this nuts? do people have better ways to achieve the stated goal?
We are talking about latency for the most part as it is the one thing measured in milliseconds. As for data placement, you may drive yourself nuts. Trust me that many have tried to optimize this in ZFS and if it were something that could be done easily, it would already be built into either the ZFS or Zpool command. ZFS file placement is not contiguous, it drops pieces all over the place. And it pulls data from usually multiple drives.
If you want better latency, run mirrors. If you want faster transfer rates, run striped mirrors. I’m just talking about HDDs alone. There are other ways to improve throughput. If you need FAST, then replace with NVMe.
That might be true for FAT and NTFS, but not for ZFS or recent Linux filesystems. They don’t just write data on the next sequential free space starting from the first LBA. The data is written in a staggered manner to reduce future fragmentation.
While the concept of writing garbage to the first 35% and then writing your static files after may seem like it would work, (and may actually work to some degree), their are other considerations.
ZFS is a COW, (Copy On Write), file system. This means it uses free space for all writes and as a last step in a write transaction group, free up any blocks that are superseded by any new writes.
Part of the COW would free up directory blocks and other Metadata in the last 2/3rds of your disk(s), which would eventually be re-used again.
One of the bigger feature requests for ZFS, is full tiered storage. I don't mean these:
- L2ARC / Cache device(s)
- SLOG / Log device(s)
- Special Allocation devices, (aka Metadata and or small block devices)
A long time ago, I worked with a real tiered file system, SAM-FS, (aka QFS). This allowed multiple tiers of caching. For example, we normally used regular disk backed by high access speed tape. All files were eventually copied to tape, however, if they remained busy, a copy was kept on disk. Any changes got copied to tape. That was based on a last change timeout value, to allow files being updated to delay writing to tape. (To avoid tape churn and shoe shinning…)
SAM-FS also supported 3 tiers, which could be:
- Fast disk / SSD
- Slow disk, (like larger SATA)
- Tape, (either high capacity or high access speed)
In any case, ZFS & OpenZFS have what they have. And don’t have preferred HDD storage locations.
Hmm, on a completely empty volume I assumed ZFS would write contagiously and sequentially from the outer tracks to the inside tracks. The reason that I thought this was a safe assumption was because of my repeated observed behavior of copying content to the disk and as it fills the transfer rate drops slowly just as you would would expect if the reason was tied to the track it is writing to. If ZFS doesn’t behave this way than I would need to explain this observation.
I had been assuming it would. This is the declining performance I see when writing to an empty volume till it is full
I understand COW, and I understand that if my proposed situation worked like I thought it would, if I modified any of the static data I would free up empty space in the slower areas. Still the more static it was, the more likely future writes would be in the prime area. That’s what I was hoping/assuming
It might, but will it? If the system decides to delete a few files, does that mean ZFS will skip those now free sectors or fill them?
I was always under the impression, and I do not know this as fact, that ZFS fills the first open sector it can find. Also, in a RAIDZ, the files are spread out across all drives and parity is included. This does not make for a straightforward recording of data.
But this is just the way I thought it worked. I am not a ZFS expert, far from it. But I would not bet that a RAIDZ will place files contiguously on the drives.
I think @Arwen has said it all best.