I’m starting to run out of space on my NAS, so I’ve been looking at how to expand my storage capacity from 3x 8TB drives in RaidZ1 to 6x 8TB drives in RaidZ1 (full specs of my system at the bottom). The question I have is: what is the best way to go about this?
Expanding my current pool with ZFS Expansion seems like the easiest way to go about it, but of course comes with the downside that the reported capacity in the TrueNAS GUI is incorrect even after using the rewrite command, which I’m not a fan of. However, if files accessed from my NAS via my Windows PC or within apps like NextCloud still report their correct file sizes (eg, I move a 10gb file to a dataset via SMB by my windows computer and the file still reports 10gb as the file size when viewing the file properties, or I upload a 10gb file to my nextcloud and it still reports as 10gb in nextcloud), I could probably live with this and hope for an eventual update to TrueNAS that fixes the incorrect reporting in the TrueNAS GUI.
Another option I’ve thought about is backing up my NAS using Cloud Sync, deleting my current pool, remaking the pool with the new drives, and then restoring from the Cloud Sync. However, I feel like this won’t be as simple as it sounds. My main concerns are that I’m sure all the data in the datasets I’ve created will be fine, but I’m not sure about data stored by some of my apps in ixVolume. I’ve also never used Cloud Sync, so I’m not sure how simple the process will be, but I know this will take a long time to complete. I just feel like this method will have a lot of unintended side effects that would make me wish I just did the ZFS Expansion and lived with it.
I could also just do a ZFS expansion, find out if I can live with it, and then look at other options if I decide I can’t, but I’d like to know what I’m getting myself into before I dive off the deep end. If there are any other methods to go about expanding my capacity that may be easier or have less side effects, or you can give more info on the concerns I have with the two methods I’ve looked at, please let me know. I can also provide more info if needed.
I suspect that the Optane M10 is intended as SLOG, which is definitively NOT useful for a raidz1 hosting SMB shares.
The key question is how you’re expanding to (at least) 6 SATA ports, and how many ports you will have in total as any “better” process will involve replicating from one pool to another.
Let’s say also that as you’re going to more drives (or larger drives) and have no local backup you should really, really consider moving up to raidz2 (or higher).
Which then necessarily involves backup-destroy-restore.
Of course they always do!
Hope not! The fundamental issue is upstream in OpenZFS, was done purposefully to ease completing raidz expansion, fixing it is expected to be of similar complexity as adding raidz expansion in the first place, and so far no one is footing the bill for development. So it’s not going to happen.
How are you planning on attaching all your drives, etc? MB seems to support 2 x M.2 slot(s) and 4 x SATA 6Gb/s ports
I’m planning on getting an HBA to connect the other two drives that won’t be able to be connected to the MB. Not sure exactly which one yet, I haven’t gotten that far in my planning, mainly because I’ve been caught up thinking about this stuff. But I do have the PCIe x16 slot free for one.
You probably don’t need the Optane for a L2ARC. You can take a look at your arc_summary statistics to get an idea.
I learned that my optane cache drive was probably worthless after I already put the system together. I ran that command, but not sure what stats I should be looking at to see if I’m getting any use out of it. I did see under L2ARC Breakdown, the hit ratio is 1.3%, so I assume that means it’s not doing much.
What apps do you have and are planning to run. It appears you have everything on a single pool.
This is my first NAS, and originally was only supposed to be for storage, but over the past few years I’ve started doing more with it. Right now, the apps I have are fireshare, immich (two of them, one is an old one I couldn’t get to update a while back and have kept installed just in case), jellyfin, nextcloud, nginx proxy manager, and some others that are easy to reinstall. Of those, Jellyfin, old immich, and nginx are using ixVolume. Everything else has their own dataset with sub-datasets for their storage. I do have everything in one pool, with multiple datasets in that pool (One for apps like new immich, nextcloud, etc. and their associated datasets, one for my Jellyfin media, and one for general storage.)
I suspect that the Optane M10 is intended as SLOG, which is definitively NOT useful for a raidz1 hosting SMB shares.
The key question is how you’re expanding to (at least) 6 SATA ports, and how many ports you will have in total as any “better” process will involve replicating from one pool to another.
I replied to SmallBarky above about optane and my plan for more SATA ports. For the optane, I learned after the fact that it probably wasn’t needed or helping. For the SATA ports, my plan is to get an HBA with at least two more ports to add to the four I have on my motherboard right now. Which HBA specifically I haven’t looked into yet.
Let’s say also that as you’re going to more drives (or larger drives) and have no local backup you should really, really consider moving up to raidz2 (or higher).
Which then necessarily involves backup-destroy-restore.
This is something I’ve been considering, but to keep costs down (especially with drives going up in price) I’d like to keep using my 3 current 8tb drives, and get three more to fill my case (something I forgot to mention in my previous post, my case only has 6 3.5” drive slots). As for going to RaidZ2, I’ve also considered that, but I think I’ll stick with RaidZ1.
Of course they always do!
Great to hear! Couldn’t find an answer in my searches so I’m glad to hear that confirmed now.
Hope not! The fundamental issue is upstream in OpenZFS, was done purposefully to ease completing raidz expansion, fixing it is expected to be of similar complexity as adding raidz expansion in the first place, and so far no one is footing the bill for development. So it’s not going to happen.
I figured that might be the case from what I read, but as long as the data per file still reports correctly in my apps and in Windows, I can live with that error in the TrueNAS GUI, or so I tell myself.
You can just reply to us in general unless you need to break out a specific reply.
You can probably remove the Optane. L2ARC and SLOG can be removed without hurting the pool / VDEV. That frees up an M.2
Apps are normally installed on a mirror VDEV of SSDs or NVMe with their own pool.
A typical setup
Boot drive for boot-pool (you might consider moving your boot to a NVMe with a USB adapter to free up onboard M.2 slot)
Mirror VDEV of 2 SSD / NVMe for APPS pool
Data Pool (whatever VDEV / pool layout you choose)
Changing to Raid-Z2 is recommended due to your drive sizes and having more redundancy. Lose two drives in Raid-Z1 and the pool is gone. Raid-Z2 allows two drives down. Raid-Z1 with a lost drive and the resilver process / time to complete is considered risky because you can’t have any more failures at that point.
One issue with adding a HBA card, like the suggested ones from the Hardware Guide in the online documentation, is they require excellent cooling air flow. Manufacturer specs 150-200 linear feet per minute over the heatsinks. They are designed for the fast, noisy server cooling setups. Some users add a fan blowing over the heatsinks.
Any LSI SAS HBA would do, basically a 9300-8i would do (or a 9400, or an older 9207/9211 but that’s getting really long in the tooth). And then cooling for it!
(You could get away with an ASMedia 11xx SATA controller without port multiplier BUT vetting the right specifications from the crowd of “no-name SATA cards from Shenzhen” is going to be a pain, the card may quite possibly come out more expensive than a refurbished SAS HBA… and the x1 thingie will fully take your sole PCIe slot anyway. Which is a long way to say that the initial mistake was a 4-port motherboard in a case with 6 drive slots; B550 can support 6 SATA ports.)
So it really was L2ARC, which is puzzling because the whole point of a Level 2 ARC is being larger than the primary ARC (i.e. RAM)… and that’s not the case here.
As for arc_summary, you first use the system for days under real conditions to let ARC and L2ARC fill up and only then run the command and look for hit ratios.
Better put a larger SSD as a single drive pool and move your apps there. Possibly even 256 MB might do, and as long as the actual data is in host mounts from the HDD pool, all you’re “risking” in the single drive app pool is the configuration of the apps—write that down to be able to recreate.
Your data, your choice…
Have a burnt-in cold spare at hand and react quickly in case of issues. ZFS is designed to keep data safe, but experience shows that when things go downhill, they can go FAST! (And then ZFS actually becomes a hindrance to saving your data after a disaster.)
If you can afford four more drives in one go, you might actually create a doubly-degraded raidz2, migrate your data to it, and then resilver twice with two drives from your old raidz1 (the third drive ends up as cold spare). CLI involved, but no raidz expansion so no botched space reporting.
I had always intended to use the sole x16 slot for a hard drive expansion card of some kind, as I don’t have any other uses for that slot at the moment or in the near future. Worst case, I think this motherboard supports bifurcation, but I’d have to check for sure. But I don’t mind using it just for a HBA. And to SmallBarky’s point above, I don’t mind adding extra cooling for the HBA either.
Originally, I only had 16gb of RAM in the system. But when I upgraded my PC from AM4 to AM5 last year, I replaced the 16gb from the server with the 32gb from my old PC.
As for moving apps to a dedicated pool, I’m not sure how I would do that, or what the benefits would be. Is the actual installation location of the app different from the different locations for data under the “storage configuration” settings (host mounts)? Could I use my 32gb optane for this app pool or would I need more storage for that?
RaidZ2 vs Z1 is something I’m still thinking about, and not something I’m locked into. I could make do with Z2, but I’d probably have to expand sooner than I’d like if I ended up going that route, but that depends on outside things that don’t need to be gone into here. Plus, I’m waiting for storage prices to hopefully drop before I do this upgrade, so I have time to think about it.
I like the sound of this, especially since it doesn’t result in misreported storage space, or me “wasting” money to back up my data to the cloud for only a month (I’d instead get another drive for a bit more money, but the money doesn’t feel wasted). As long as I can keep my apps working fine and the associated data for them intact (maybe even do the single/mirrored SSD for apps thing at the same time), this could be the way to go for me. And I assume if I really wanted to stay on raidz1, this could be adapted to work with that as well?
The APU supports x8x4x4; check for the board.
There are risers to hold one half-height PCIe card (x8) and a pair of M.2 in a full height slot, so that could be your HBA and a mirrorred app pool.
TrueNAS lets you move the app dataset across across pools. The benefits of a dedicated SSD pool are faster I/O and more efficient handling of small files compared to raidz.
That’s the whole point of host mounts: Have the app itself in one place (ideally a SSD), and the actual data somewhere else where there’s enough space and an appropriate backup policy.
Yes, but you would then need five disks in one go (total: eight) to do it reasonably safely (two copies and/or redundancy at any point).
Or you do it with seven drives, reusing one of your current drives to create the new pool, which means that at some point in the process your data will be at risk with just one copy and no redundancy—which I would not advise.
And this is done by going to configuration > choose pool in the app tab? That’s a lot easier than I thought if so.
I did some thinking last night and RaidZ2 is what I’ll end up going with. Is there any tutorial you’d recommend on how to do the “doubly-degraded raidz2” method you mentioned?
Of course, “redundancy vs. capacity” is the name of the game…
The solution would be to aim for drives larger than the current 8 TB, say 12-16 TB, as a starter for the new pool and progressively replace the 8 TB as you. Initial raw capacity is 4*8 = 24 TB for the mixed 6-wide raidz2 pool, and after replacing the last 8 TB you have one button to press to extend to 4*12 (or whatever). No weird space reporting.
This will be your only way to increase capacity once you have reached the maximum number of drives.
The outline is:
optionally unplug the old pool to prevent mistakes while dealing with the CLI
Create the degraded pool.
Attach 4 new drives, format them on the CLI to mimick what TrueNAS would do (this is where some research is needed), create 2 sparse files of appropriate capacity and then create your 6-wide raidz2 with the sparse files and the UUIDs of the real drives zpool create newpool raidz2 /home/admin/sparse1 /home/admin/sparse2 UUID1 UUID2 UUID3 UUID4
(plus probably a few options for compatibility such as -R /mnt)
then offline the sparse file devices (and delete these files for good measure)
Once you can export and reimport the degraded pool from GUI, you’re done dealing with the CLI.
plug back oldpool (you need to attach 7 drives at this point, but it is acceptable that one lies outside of the semi-open case, with data and power cables coming out…)
Replicate your data from oldpool to new pool.
Shutdown and detach two drives from oldpool, which you keep as (degraded) offline backup.
Take a deep breath and reboot. Degraded newpool should come up; replace one of the missing devices (sparse files) with the remaining old 8 TB drive.
(At this point, you have two non-redundant copies of your data. Once resilver is complete, newpool has one degree of redundancy.)
Power down, bring in another old 8 TB drive, reboot and replace the other missing device with the 8 TB drive (which then destroys oldpool beyond recovery).
Rename newpool as oldpool according to the Resource.
Wipe the last remaining old 8 TB for use as cold spare.
I can’t lie, a lot of the “create degraded pool” point went over my head lol, but I tried to do some research to learn my way around what you’re saying.
For creating the degraded pool, I found this guide. I assume I’d follow that guide and stop after step 15, then start following your steps from the third bullet point (“plug back oldpool” onwards).
I believe I can use the “replication tasks” function to copy over all data that is on oldpool over to newpool, so my app’s data should move over seamlessly once newpool is renamed?
Replacing the sparse files with the oldpool disks seems as simple as following this documentation, and selecting “force” and deselecting “preserve disk description”.
That sounds all correct.
Do not hesitate to experiment with pool creation on the CLI using a test setup, possibly TrueNAS in VirtualBox or another desktop hypervisor (as long as no real data is comitted there, it’s all fine).
@etorix I hope you dont mind me asking for some more help here. I’m going through the guide I linked, but I’m currently having an issue at step 9. I cannot create a sector starting with 2048, I am only able to start it with 32760.
Should I just continue with the defaults fdisk is suggesting for starting and ending sectors, or should I use a different tool (parted) to set the start and end sectors like how the guide states I should? If I do that, it warns me The resulting partition is not properly aligned for best performance: 2048s % 32760s != 0s, so I want to be sure before fully committing.
EDIT: I tried to do this with a second disk, but in parted, using the values I got in step 4 for this drive gives me the error The location 23433578495 is outside of the device /dev/sdc., so I don’t think I can use that method either… Furthermore, this only seems to apply to the 12tb drives I have, the temporary 8tb drive (7th drive) I can set the starting and ending sectors “correctly”, but when doing this I get a warning: Partition #1 contains a zfs_member signature. Do you want to remove the signature? [Y]es/[N]o:, so I’m honestly not sure what I should be doing lol
Use whatever tool you’re most comfortable with.
In doubt about how TrueNAS would actually format disks you can create a (smaller) pool in the GUI (layout doesn’t matter), and then read the partition table to copy the values.
I think that is what I’ve been doing with the steps I’ve been following but I’m quite out of my depth here. I create a pool with the disks, run sudo fdisk -l /dev/sdX and note the starting and ending sectors, which I then use later in fdisk to create the partition. The problem is I cannot use those values in fdisk, and it recommends different ones.
Would there be any downside to just using what fdisk recommends for the starting and ending sectors? As far as I can tell the only difference is that there would be more sectors than what TrueNAS has created, but it seems like that might be by design. So should I use the recommended starting sector (32760 instead of 2048) and use the ending sector I got from the fdisk command earlier?
On pool creation, TrueNAS sets 2 GB aside as safety buffer in case you have to replace the drive with one which would happen to be a lttle smaller. (This is very relevant for SSDs, possibly less for HDDs now.)
I suppose that either choice work. Personally I would override fdisk.
