I want to move my setup from unraid to truenas and am looking how to best do this.
My current server has basically 3 disks: 2x data (equal size) and 1x Parity (different size than data disks), additionally I have a pool of 2 mirrored 1TB NVME drives for caching.
When switching over I am wondering how to best set it all up.
Current thoughts ZFS Mirror for the two data disks. I can offload the data to a usb disk if and then copy it back that way if that is an option.
What I am unsure of is what to exactly do with the 2x 1TB NVME drives. Until now I have used those for apps (e.g. docker volumes) as well as caching read/writes on unraid.
I have since moved most of my dockers to a different minipc and want this server to be what truenas does best, datastorage.
Would it make sense to also have caching on truenas and if so how to set it up? Does that also have to be mirrored? Or would it make sense to have a small app Pool on the NVMEs, might run a few dockers directly on truenas for easier access to the data.
Basically trying to do an in place switch of the system without loosing my data and in a way it still makes sense for my hardware.
I realize I might have 1 disk left over if I go this way (former parity drive).
In terms of hardware I do have 32gb of ECC Ram, it’s an older Xeon based system.
I would agree with you that TrueNAS is most mature and stable in the pure NAS space and it does this extremely well. TrueNAS’ implementations in the container space has been in a constant state of flux for years, which means it is less mature and has greater ongoing maintenance effort, however that means you need to weigh up the trade-offs of this flux against the cost savings and performance gains of running everything on the same box and the extra effort of maintaining separate infrastructure for VMs & apps.
HDD pool
The issue with using the 2x equal sized HDDs for a mirror is that you will only have half the usable storage as you currently have on UnRaid. But if that is enough for you then great.
The alternative would be to use the 3x HDDs for a RAIDZ1 pool which is as similar as you can get to your current UnRaid configuration - and with pretty much equal useable storage (and equal wasted storage on the larger HDD).
Read Caching
The primary read cache in ZFS/TrueNAS is called ARC (Adaptive Read Cache) which is in main memory. You may therefore need more memory for TrueNAS than you did for UnRaid - but 32GB is more than you will need if you don’t run VMs.
You can add a 2nd-level cache using SSD/NVMe (L2ARC) but it is usually far more effective and cost-effective to add more memory than to add an L2ARC. In your case 32GB is more than enough for ARC and L2ARC will probably not do anything more to improve performance still further.
Write caching
The primary write cache in ZFS/TrueNAS is also main memory. Normal sequential writes of files are (or should be) asynchronous writes, which means that they are stored (and grouped) in memory and written out every 5 seconds as a transaction group (TXG). The TXG guarantees that (on correctly configured hardware) the ZFS file system remains consistent in the event of a sudden failure (crash or power failure) however up to 10s of data updates can be lost.
Sequentially written files remain consistent but virtual disks / zVolumes / iSCSI and randomly written files (e.g. database files) can lose consistency - and for these special cases you need to use synchronous writes which have a significant performance penalty especially on HDDs which do seeks - so you either need to have the data on SSD or use an SLOG SSD device to offload the performance penalty writes.
In addition, the random accesses that require sequential writes also tend to be small 4KB blocks, and you normally need to use mirrored pools to avoid read and write amplification.
Your 2x 1TB NVMe
For the 2x 1TB NVMe disks, IMO you have two choices:
Recommended - Use them for a separate mirrored apps pool for your very active data, virtual disks / synchronous-writes data etc. i.e. for whatever containerised apps you want to move back to TrueNAS and for any virtualised disks you want to expose to VMs running elsewhere; or
Alternative - Use them as a special allocation (metadata / small files) vDev on your HDD pool. This needs to be pre-tuned before you start to write to the pool because there are few if any tools to move existing data on HDD to the special allocation vDev, and for this reason this isn’t the recommended option.
First of all Thank you so much for your reply. It is very informative and helped me gain a better understanding of a few concepts and what I might want to do.
Now a little bit more specific:
This does make a lot of sense to me and seems like that would absolutely be the way to go and make good use of these disks!
I was looking at this as well and in terms of usable space this would definitely be the better option I agree. I was just unsure about RAIDZ1 in terms of upgradeability. What happens if I run out of space? Can I just replace a drive and resilver?
Maybe I need to add that my current setup is limited in terms of space so with the case closed I can have 3xHDDs I do have power and Sata for a 4th one in case I would need to hook it up temporarily which I may need to use that one for the actual boot pool (?) for truenas itself which would leave me with exactly the 3 HDDs for RAIDZ1. Just not knowing any better I wasn’t sure if I needed to replace drives with a larger one where that would leave me.
Lastly, a practical question: Any sense in creating a ZFS pool on the NVMEs in Unraid prior to moving? Would that make things easier for the move or would it be best just to offload all the data and start creating all pools etc in Truenas and then move the data back?
And yes, I do have backups.
One thing that I hope you’re already aware of, is that TrueNAS isn’t as flexible with different-sized disks or expansion as Unraid is. ZFS did add an ‘expansion’ feature to add disks to existing raidz, but it’s not a feature I’ve used and I’ll leave it to others.
On ZFS, all disks within a vdev are treated as the size of the smallest disk. If you have two 8TB disks and a 20TB disk, all will be treated as 8TB. Now, if you replace the two 8TB disks with 20TB disks later and let it resilver, ZFS will automatically treat all disks as 20TB and capacity will expand. Some people increase size of a pool by replacing the disks with larger ones.
For making a ZFS pool with the nvme now, that’s up to you. As long as they don’t create the pool with higher feature flags than TrueNAS (doubtful), you should be able to import them as-is on TrueNAS with their existing data. I’d use the default 128k recordsize for an Apps pool.
Yes - this is one option. If you replaced (one-by-one) each of the smaller drives with another of your larger drive and resilver, then when all drives are the same larger size you can expand to use the full drives giving you 2x drives worth of useable space.
Or (if you had more ports - which you don’t) you could do RAIDZ expansion by adding another drive which is at least as big as the smallest existing drive in the vDev.
Yes - you will need a small SATA SSD as a boot drive.
Maybe. The NVMe pool will either import into TrueNAS or it won’t - it should work so it is definitely worth trying, but you will need an alternative plan if it doesn’t work.
Perfect, that is exactly what I wanted to hear. Well then maybe RAIDZ1 is an option. I originally just thought it wasn’t that is why I figured I’ll go with mirror.
Thank you for pointing that out. And yes, I am absolutely aware of that. That is basically why I was asking because I wanted to understand my options better.
Thank you both so much for your comment! Very helpful!
I made the switch, created a raidz1 Pool with my 3x HDDs and imported the cache pool 2x NVME that I migrated to in Untaid prior to switching and chose that as my appdata pool in Truenas.
If anyone is interested importing the pool from unraid worked without any issues. It was a mirrored pool of 2x 1TB NVMEs that I had used in unraid for my appdata.
Now I am just looking to see how I connect to my UPS (which is already connected to another device on the network). Additionally I am trying to see if RSYNC is available to me to transfer my files easily over the network onto my new Truenas Server or if I would be able to just connect and mount a XFS formatted USB drive.
Short Answer:
To be honest mainly licensing cost and the fact that I don’t necessarily need the convenience features.
Long Answer:
I am one of the unlucky ones that thought about unraid for a long time not having the hardware, then when I finally had the hardware (an older Xeon based system with a bunch of ECC Ram) I started with unraid but It was after they changed to the new licensing model. Overall it is a really good system, but… I just don’t see myself shelling out $250 for a lifetime license at this point especially when I could also do it as baremetal Debian server. I have gotten pretty comfortable with the command line over the years, just new to ZFS at this point.
As much as I love the idea of a flexible array I do not necessarily need it. I have all drives of the same capacity and for the number of drives I have, it isn’t overly crazy to setup ZFS, which I love the data integrity features.
Could have done all of that on unraid as well now, but that would have come at the licensing expense. My storage needs are fairly humble, but it is critical data to me (mainly family pictures, memories).
Truenas gives me what I need in terms of storage and I can do what I need regarding Apps on Docker which I migrated my dockers to a small mini pc, older hardware with SSD Storage and Proxmox.
I more or less decided to separate Apps and Storage only and for Storage Truenas gives me what I need at basically no cost compared to unraid.
Not sure if that makes sense to you. Unraid is not a bad system at all but for my situation with the hardware and knowledge that I have truenas seems to be the better and more cost efficient solution.
Understood, took me second to find it in the Data Protection Tab! Perfect! Still learning.
I do have one more question: Can I mess with the files in /etc/nut directly through SSH as opposed to using the webgui (that honestly makes it harder for me at the moment) because I could just copy the files for /etc/nut from another system that I already have working.
Overall total, very happy with my switch. The speed is incredible compared to before. The only issue I have had by pulling data onto the Truenas using RSYNC was that I would not be able to preserve file times with rsync. I somehow always got a permission error that I am not allowed to set times.
Haven’t been able to solve that. Other than that it has been all incredible so far.
