Hi,
I want to move my setup from unraid to truenas and am looking how to best do this.
My current server has basically 3 disks: 2x data (equal size) and 1x Parity (different size than data disks), additionally I have a pool of 2 mirrored 1TB NVME drives for caching.
When switching over I am wondering how to best set it all up.
Current thoughts ZFS Mirror for the two data disks. I can offload the data to a usb disk if and then copy it back that way if that is an option.
What I am unsure of is what to exactly do with the 2x 1TB NVME drives. Until now I have used those for apps (e.g. docker volumes) as well as caching read/writes on unraid.
I have since moved most of my dockers to a different minipc and want this server to be what truenas does best, datastorage.
Would it make sense to also have caching on truenas and if so how to set it up? Does that also have to be mirrored? Or would it make sense to have a small app Pool on the NVMEs, might run a few dockers directly on truenas for easier access to the data.
Basically trying to do an in place switch of the system without loosing my data and in a way it still makes sense for my hardware.
I realize I might have 1 disk left over if I go this way (former parity drive).
In terms of hardware I do have 32gb of ECC Ram, it’s an older Xeon based system.
Any help is greatly appreciated.
Thank you!
General comment
I would agree with you that TrueNAS is most mature and stable in the pure NAS space and it does this extremely well. TrueNAS’ implementations in the container space has been in a constant state of flux for years, which means it is less mature and has greater ongoing maintenance effort, however that means you need to weigh up the trade-offs of this flux against the cost savings and performance gains of running everything on the same box and the extra effort of maintaining separate infrastructure for VMs & apps.
HDD pool
The issue with using the 2x equal sized HDDs for a mirror is that you will only have half the usable storage as you currently have on UnRaid. But if that is enough for you then great.
The alternative would be to use the 3x HDDs for a RAIDZ1 pool which is as similar as you can get to your current UnRaid configuration - and with pretty much equal useable storage (and equal wasted storage on the larger HDD).
Read Caching
The primary read cache in ZFS/TrueNAS is called ARC (Adaptive Read Cache) which is in main memory. You may therefore need more memory for TrueNAS than you did for UnRaid - but 32GB is more than you will need if you don’t run VMs.
You can add a 2nd-level cache using SSD/NVMe (L2ARC) but it is usually far more effective and cost-effective to add more memory than to add an L2ARC. In your case 32GB is more than enough for ARC and L2ARC will probably not do anything more to improve performance still further.
Write caching
The primary write cache in ZFS/TrueNAS is also main memory. Normal sequential writes of files are (or should be) asynchronous writes, which means that they are stored (and grouped) in memory and written out every 5 seconds as a transaction group (TXG). The TXG guarantees that (on correctly configured hardware) the ZFS file system remains consistent in the event of a sudden failure (crash or power failure) however up to 10s of data updates can be lost.
Sequentially written files remain consistent but virtual disks / zVolumes / iSCSI and randomly written files (e.g. database files) can lose consistency - and for these special cases you need to use synchronous writes which have a significant performance penalty especially on HDDs which do seeks - so you either need to have the data on SSD or use an SLOG SSD device to offload the performance penalty writes.
In addition, the random accesses that require sequential writes also tend to be small 4KB blocks, and you normally need to use mirrored pools to avoid read and write amplification.
Your 2x 1TB NVMe
For the 2x 1TB NVMe disks, IMO you have two choices:
-
Recommended - Use them for a separate mirrored apps pool for your very active data, virtual disks / synchronous-writes data etc. i.e. for whatever containerised apps you want to move back to TrueNAS and for any virtualised disks you want to expose to VMs running elsewhere; or
-
Alternative - Use them as a special allocation (metadata / small files) vDev on your HDD pool. This needs to be pre-tuned before you start to write to the pool because there are few if any tools to move existing data on HDD to the special allocation vDev, and for this reason this isn’t the recommended option.
(I hope this helps.)
2 Likes
Solid reply, and I agree with using it as a mirrored apps pool.
First of all Thank you so much for your reply. It is very informative and helped me gain a better understanding of a few concepts and what I might want to do.
Now a little bit more specific:
This does make a lot of sense to me and seems like that would absolutely be the way to go and make good use of these disks!
I was looking at this as well and in terms of usable space this would definitely be the better option I agree. I was just unsure about RAIDZ1 in terms of upgradeability. What happens if I run out of space? Can I just replace a drive and resilver?
Maybe I need to add that my current setup is limited in terms of space so with the case closed I can have 3xHDDs I do have power and Sata for a 4th one in case I would need to hook it up temporarily which I may need to use that one for the actual boot pool (?) for truenas itself which would leave me with exactly the 3 HDDs for RAIDZ1. Just not knowing any better I wasn’t sure if I needed to replace drives with a larger one where that would leave me.
Lastly, a practical question: Any sense in creating a ZFS pool on the NVMEs in Unraid prior to moving? Would that make things easier for the move or would it be best just to offload all the data and start creating all pools etc in Truenas and then move the data back?
And yes, I do have backups.
One thing that I hope you’re already aware of, is that TrueNAS isn’t as flexible with different-sized disks or expansion as Unraid is. ZFS did add an ‘expansion’ feature to add disks to existing raidz, but it’s not a feature I’ve used and I’ll leave it to others.
On ZFS, all disks within a vdev are treated as the size of the smallest disk. If you have two 8TB disks and a 20TB disk, all will be treated as 8TB. Now, if you replace the two 8TB disks with 20TB disks later and let it resilver, ZFS will automatically treat all disks as 20TB and capacity will expand. Some people increase size of a pool by replacing the disks with larger ones.
For making a ZFS pool with the nvme now, that’s up to you. As long as they don’t create the pool with higher feature flags than TrueNAS (doubtful), you should be able to import them as-is on TrueNAS with their existing data. I’d use the default 128k recordsize for an Apps pool.
Yes - this is one option. If you replaced (one-by-one) each of the smaller drives with another of your larger drive and resilver, then when all drives are the same larger size you can expand to use the full drives giving you 2x drives worth of useable space.
Or (if you had more ports - which you don’t) you could do RAIDZ expansion by adding another drive which is at least as big as the smallest existing drive in the vDev.
Yes - you will need a small SATA SSD as a boot drive.
Maybe. The NVMe pool will either import into TrueNAS or it won’t - it should work so it is definitely worth trying, but you will need an alternative plan if it doesn’t work.
ok, I’ll try that.
Perfect, that is exactly what I wanted to hear. Well then maybe RAIDZ1 is an option. I originally just thought it wasn’t that is why I figured I’ll go with mirror.
Thank you for pointing that out. And yes, I am absolutely aware of that. That is basically why I was asking because I wanted to understand my options better.
Thank you both so much for your comment! Very helpful!
