Yes, there is no tiered storage integrated inside ZFS (and it’s a pity, really). But, you can sort of do that by yourself with a bit of scripting abilities… ie, running a cronjob every day that copies everything you have written to an SSD pool to your HDD pool, and then erases the files on the SSD pool.
Or, just let the drives spin up and down a few times per day.
I’d script that VERY carefully in case the user accidentally starts using folder names that are identical to the contents on the HDD pool. So much to potentially go wrong there.
yeah some tiered caching wouldn’t be bad… but I understand that the PROs users don’t really let their HDDs spin down at any time (maybe in 10 years it won’t really makes sense if SSDs keep growing in size and therefore energy consumption won’t be really an issue at all 
) . Anyway… I am more of a windows guy, so scripting is not my thing and as Constantin said, maybe you forget a folder and something goes wrong… so I guess more pain than gains.
I’d propose my structure of HHDs/SSDs and then you can tell me your thoughts about it:
For the caching “issue” I’d propose to leave HDD with spin down automatically when not in use as far as possible OR I found a “good” deal for 10x120GB sata SSD and maybe put 1 ssd disk as Slog on each pool? but yeah, if the writes happens every 5 seconds I don’t know if it really makes sense (maybe you can give me some insights)
For torrents I will keep the seeding files in the Apps&Vms pool and as soon as seeding is finished i’ll let sonarr and radarr move the files in the media pool.
What do you think so far? 
I’d monitor that very carefully to see if the HDDs actually get to rest for long periods of time. If they don’t, then there is no point to spin them down in the first place. Spin-up strains the motors, so it should be limited, if possible. There is a spin-down resource over in the old forum that I’d take a look at re: guidance.
As a intermediary to SSD, consider Helium-filled HDDs, which tend to consume less power than Air-filled HDDs.
Most of the time my HDDs are spun down, because either I watch movies/tv series at night and the rest of the day the server is mostly idling and downloading new movies tv series. I access files during the day, but not so not many spin up and downs I would say, is more a casual thing. During the night instead I will develop on VM and stuff like that. Are there any helium filled HDD with 6TB or 4TB? that’s the sizes of HDD I can do raidz with at the moment, instead of buying all new HDDs
It won’t take ten years. Find me a 60 TB HDD today:
Ahahahah, those SSDs are very cool but the price… Almost 4000 dollars for the 44TB version (which is my entire current server storage size in 1 single disk!! hahahah). Hopefully also us, humans who don’t work in a server environment will find SSDs with cheaper prices and bigger sizes in a few years. I think that reaching 10TB for 500 euros per SSD would already be a good sweetspot for many (if only they don’t bitrot in the long run). What do you think about the storage configuration I’ve made a few posts above? Does it look correct?
Either use one or read Highly Available Boot Pool Strategy | TrueNAS Community.
Better to use it for block storage with iSCSI (the only reason I believe you would passthrough an SSD).
You can have these in the same pool.
Likely it’s not going to work. I would set at the very least a 60 minutes waiting time before spinning down the drives… better if 120.
A SLOG is not a write cache. You likely do not require a SLOG.
That’s what I do as well, it prevents fragmentation in the HDD pool.
If it fails, yes.
Block storage with iSCSI would be ideal.
Finally, I recommend running @joeschmuck’s Multi-Report.
Is the scenario with iSCSI supported by truenas scale? And can I “split” the device in 2? I mean I assign 500 GB to a VM and the rest 500GB to another VM?
I believe Unraid supports ZFS now but I’m not super familiar on how they implement it. Just an FYI.
I learnt a lot with Unraid and I am still thinking it is a valid product and the community is also great but it is very community dipendendent product… Unraid is “supporting” ZFS… but replication, snapshotting etc is still not implemented and most of the functionalities/plugins etc are running thanks to indipendent developers not because of Unraid and this is a little concerning… that’s why I am thinking of moving into a more serious project like truenas where the core of the project at least is entirely developed by company developers
Right, the CORE of it… pun (and jab) totally intended.
Just another question regarding the migration. Since I have 2 Windows VM I would like to pass a the same GPU… I have the iGPU of my intel CPU and is it possible to pass it the VM in SR-IOV mode?
Two parts to your question:
The answer to 2. is “yes”. 1. is trickier.
Conceptually this is possible, and as I understand it Nvidia enables it for their professional GPUs. Intel is currently working on this for Linux, but the work is not complete as I understand it. I think laptop GPUs have this mostly working, but not desktop GPUs.
This is interesting… I just tested today on Unraid server this functionality (brought up by a user… as usual on Unraid ) and for a while it worked… for a while I had the same intel iGPU active on those 2 Win11 VMs at the same time… so I was kinda curious if this scenario is currently supported on truenas (not that it is super important but it would be a nice feature). So my usage would be quite simple… like while watching a youtube video inside the VM I won’t bog down the CPU… nothing related to video editing or stuff like that
But as you said @ericloewe it seems like it isn’t supported THAT well from GPU companies as well… so I guess I’ll stick with VM with cpu as gpu
Hey, if it worked on Unraid, it’s worth a shot with TrueNAS. Give it a try and let us know, that the sort of thing that would make my life easier.
From my looking into this, it appears that Nvidia actually has it working, but locks it behind licenses, but there are vGPU unlocks.
Meanwhile, Intel has switched from their vt-g extensions to just using SR-IOV, but not sure where that is up to, and integrated graphics pass through requires chipset AND bios support. Which a lot of Supermicro boards do not have.
Since all my servers 1) do not have integrated gpus and 2) don’t support the pass through anyway, I haven’t bothered pursuing that path further.
I’m looking into the Tesla/quadro nvidia thing currently for the same reason, I want to run multiple windows VMs, and without a gpu the rdp perf is not good enough
Yeah newer Intel IGPU use SR-IOV ( I think from 11th gen onwards) but this thing seems so new that I guess Truenas will take a long time before integration (IF it will ever be integrated)
