Hello. Rather unexperienced user here, but have had my HP-z800 freenas workstation running as a home server for documents, photos etc. for some years now. It started with a steep learning curve with no experience from Linux, Unix, RAIDs or nothing. It has done me good, and from my starting point I have learned a ton on the way. Problem is without more use of the stuff I learn, I also forget a lot on the way.
My problem:
I have a pool set up with 6 disks, where 3 pairs of 3+4TB (or 4+4TB) disks are mirrored 3-fold, so my pool capacity is now ~7TB. I have over the years replaced most 3TB disks with 4TB disks, and have now only one 3TB left in the pool. I ordered a new 8TB disk thinking that I could replace one pair of 3+4TB with the new 8TB and thereby save one disk slot, reduce heat / power consumption and increase the pool from 7TB to 8TB in the server. I am now unsure on how to best proceed with this plan to not dig myself into a deep hole. I am no longer sure if I did a striped mirror or a mirrored stripe.
My zpool status look like this:
SCRUB
Status: FINISHED
Errors: 0
Date: 2024-09-01 05:00:19
Name
Read
Write
Checksum
Status
/mnt/Data1-RAID1
0
0
0
ONLINE
MIRROR
0
0
0
ONLINE
da0
0
0
0
ONLINE
da4
0
0
0
ONLINE
da3
0
0
0
ONLINE
MIRROR
0
0
0
ONLINE
da1
0
0
0
ONLINE
da6
0
0
0
ONLINE
da5
0
0
0
ONLINE
My disks list looks like this (before maing any changes):
Name Serial Disk Size Pool Disk Type Model
da0 WD-WCC4E1CFES0F 3.64 TiB Data1-RAID1 HDD ATA WDC WD40EFRX-68W
da1 WD-WCC4N1REYFXJ 2.73 TiB Data1-RAID1 HDD ATA WDC WD30EFRX-68E
da3 WD-WCC7K3ACZJTX 3.64 TiB Data1-RAID1 HDD ATA WDC WD40EFRX-68N
da4 WD-WCC7K4DLY5VJ 3.64 TiB Data1-RAID1 HDD ATA WDC WD40EFRX-68N
da5 WS24MX7X 3.64 TiB Data1-RAID1 HDD ATA ST4000NE001-2MA1
da6 WS24MLAC 3.64 TiB Data1-RAID1 HDD ATA ST4000NE001-2MA1
da2 11076500320009980004 111.79 GiB freenas-boot SSD ATA Corsair CSSD-F12
da7 4C531001610907123555 28.64 GiB freenas-boot UNKNOWN SanDisk Ultra Fit
da8 4C530001260908100104 28.64 GiB freenas-boot UNKNOWN SanDisk Ultra Fit
I would like to remove da1 (3TB) and whichever disk it is striped with (where/how do I see which disk that is??) and replace those two with the new single 8TB disk. Would that be possible, or will I have to start from scratch with a new pool setup?
Sorry for the noob qustions, and if I have not pasted information in the best/correct format. Greetings from Norway!
Not possible. You fundamentally misunderstand the geometry here: Drives are not striped across vdevs (3-way mirrors); rather da0, da4 and da3 are full copies of each other (mirror), da1, da6 and da5 are full copies of each other, and these two mirrors are then striped together.
This is the equivalent of RAID10 (RAID0 of RAID1=stripe of mirrors), not RAID01 (mirror of stripes, not allowed by ZFS).
You can replace da1, the last 3 TB, by your 8 TB drive and get one TB of extra capacity (2*4 TB). To unlock the full capacity of your new drive, get two more 8 TB drives (or larger) and replace da5 and da6; that will give 4+8 TB.
Thanks for the quick reply and for being patient with my lack of knowledge.
So if I want to reduce the amount of disks, I would have to start over,setting up a new pool? As I am unexperienced, I wanted a 3way backup where all data would be easily accessable on remaining disk if I mess up something. I have seen that the biggest risk for my raid is not disk lifetime but the risk of me doing something stupid… I also have a 4th remote backup just in case. For a home server this might be overkill x2 but I didn’t want a setup where a messup during replacement of a failed disk could cause total loss.
I might for now just replace the 3TB disk with the 8TB, but if I consider starting over, would you suggest a similar setup or is that just stupid?
Thanks again.
No. Mirrors are flexible, you can remove drives and “downgrade” to 2-way mirrors any time you want. But as drives get larger, 3-way is what I would run myself; losing a drive in a 2-way puts too much data at risk and takes too long to resilver.
What you could also do to reduce the number of drives is to go for raidz2: A 4-wide raidz2 has the same capacity and resiliency as your two 3-way mirrors with 50% less drives. But it is less flexible, it has less IOPS, the change does require a rebuild and full restore, and the point with raidz2 is rather to run it at its safetey/capacity sweet spot of 6- to 8-wide with a set of large capacity drives (16-20 TB should have the best price per TB now).
OK. I will need to think this through before I can do much.
Another problem is my 3TB remote backup disk is now full, so I was hoping to free up 3+4TB and use the 4TB as my new remote backup until my data reaches 4TB. Instead I could order a new 4TB or 8TB backup disk.
3 more new 8TB disks would allow a new 3x 8TB mirror + 8TB remote backup, but that would be some another 650-750EUR I don’t really have available now, and I would be left with 6+1 good 3/4TB disks with no use.
I might just use the new 8TB as remote backup for now, and leave the rest as is until I can set up 3+1 8TB.
Another stupid question; if I can reduce the 3mirror to a 2mirror (and thereby remove 2 disks), can I then set up the 8tb as a new, individual pool that then is mirrored or a hot backup from the first pool, giving me more or less what I first wanted with 3x redundancy from 5 hot disks? Data traffic is really low so speed might/should not be an issue. I could even run a cron job sync to update the hot backup every 24h?
Yes, you could have a single-drive pool and replicate to it as often as you want (better than resync on cron job). But this single drive pool would not be able to correct for read errors (“data rot”) on the drive, and being in the same enclosure it would have limited value as backup. Not much gain overall compared with 6 drives in 3-way mirrors.
Another way to reduce the number of drives would be to go for much larger drives—whatever comes better as NOK per TB, which I expect to be well over 8 TB in capacity. Spare for as long as it takes and get two, much preferably three of these. Replace the drives in one of your mirror vdevs, and then remove the other vdev from the GUI: This will migrate all data to the big mirror, eventually leaving you with a single vdev. (Mirrors are flexible!)
Thanks. I used to take a lot of high mpx raw photo, so I gained 2-3 TB data rather quickly some years ago. I don’t take much photos now and I believe 3x 8TB mirrored disk space would be enough for many many years still.
I am not really looking for improved safety over 3x mirror, but was hoping 2x mirror + 1x hot backup + 1x remote backup would equal about what I have today with 3x mirror + 1 remote backup.
Yes, I might rather quickly save for 3x bigger disks, but next question is if I really want to prioritize that money on getting more space that i wont need for many many years to come. I guess I was too quick on the trigger when I ordered the 8TB disk instead of just ordering a new 4TB remote backup disk.
I have learned a lot now. Thanks.
