What's the safest way to transition from CORE to SCALE without losing data stored on pools?

TrueNAS General
,
1

I’m planning to transition from CORE to SCALE once the new version is out and stable. My major concern is that there was a thread reporting data loss, which terrifies me: Pool Unavailable after upgrade from CORE 13 to SCALE 24

I’m ok losing jails and vms and my default configuration, I can recreate it manually, however I don’t want to lose any files in my pools.

My setup is currently:

  • MARS, a pool with 2 hdds, one set to mirror the other
  • VENUS, a pool with 2 hdds, one set to mirror the other
  • JUPITER, a pool with 2 hdds, one set to mirror the other
  • URANUS, a single disk pool with data that’s ok if lost
  • EARTH, a single disk pool with data that’s ok if lost

I also have 1 backup drive for MARS, VENUS, EARTH and JUPITER. The data is backed up using the UI replication task.

  1. What’s the safest way to upgrade to SCALE, is it doing a fresh installation on a new SSD and importing the pools safer than doing the “sidegrade”?
  2. If I import a pool in SCALE and things don’t workout, can I import the pool back in CORE?
  3. (crazy thought) Should I disconnect all the replica drives, import the pools in a degraded state in SCALE and once I verified everything is fine reconnect the replica drives?
  4. I will definitely use “jails” (jailmaker or sandboxes, not sure yet). Should I make a new pool, made of a single SSD, to dedicate to running software, while keeping data on HDDs?

Finally, I have a concern for my pool MARS. After a cleanup of snapshots, I ended up in a state where I cannot perform replication of auto- snapshots and francesco- snapshots, however I was able to work around it using a new naming scheme and replicating only that snapshot (francesco-fix1-). Could this be a problem once I start importing the pool in SCALE? More details about the issue are on this topic: Cannot replicate pool to brand new pool. Source pool have some problems with snapshots?

I have the option of copying all the data on MARS onto JUPITER (which is a way bigger), but I’d rather not to.

2

I highly recommend following the Preparing to Migrate guide available in the documentation. One key item is to make sure you follow upgrade paths, not sure which version of CORE you are running, but make sure you are at a valid migration version before starting.

1 Like
3

Indeed, I reviewed it yesterday night, but it doesn’t clarify if one of the two migration paths is safer and left me with a couple of questions (some specific to my setup, unfortunately)

4

First to address your specific questions:

    1. & 2. I haven’t used CORE and haven’t done an upgrade so I am not qualified to answer.
    1. No.
    1. Yes and No. Putting apps and their primary data on SSD is a good move (e.g. keep your Plex media on HDD, but put the Plex app and the Plex metadata on SSD). A mirror would be better, but a single drive is OK because you can always replicate it to HDD as a backup. My advice would be to use TrueNAS Charts where an app exists, and “Jails” with Dockage where they don’t.

My other comment are not really about migration, but rather about pool design.

  1. I assume when you say “one set to replica the other” you actually mean “ZFS mirrored”. If this assumption is wrong then please explain what you mean.

  2. There may be a better design than mirrors and single disks all in separate pools. You don’t say what size the disks in your pools are, nor what your use cases are, so we cannot tell whether RAIDZ might be possible or not. However, if we assume that your pools are all different sizes, then you might well be better off combining all your mirrors into a single pool.

If you want to give more details about your disk sizes (and precise model numbers) and space used, we might be able to suggest a better pool layout together with a migration strategy. Probably best to migrate from Core to Scale first, let it stabilise and then migrate your pool design.

5

Sorry, I definitely meant mirror and not replica. I’m sleepy today.

Regarding the pool design, I’m open to hear it, but I don’t think I’m open to RAIDZ, I considered that for a while, let me explain:

  • MARS is 4TB, almost full (75%)
  • VENUS is 4TB at 45%
  • JUPITER is 14TB at 13%

Content:

  • MARS has jails + media files
  • VENUS has family photos (very important), documents and media files
  • JUPITER has media files so far (and it’s mostly empty)
  • EARTH has backups of other computers
  • URANUS has throwaway data, not really relevant

The reason why I didn’t go with a RAIDZ are the following:

  • I don’t trust my skills as a TrueNAS admin. Confirmation of that is how I messed up some data on MARS
  • Turns out I get very stressed with backups, having a full mirror makes me way more confident if something goes wrong at the hardware level
  • I purchased the HDDs over a very long timeframe (~6 years), I didn’t really know I was going to buy a 14TB drive
  • I really appreciate the simplicity of saying “if something goes wrong, clone the HDD”, without having to deal with multiple HDDs. I can do a raw clone from any system and it would be fine
  • Read performance is the only relevant one given the large amount of media files
  • I have 24 HDD slots on my server and currently use 8

I’m open to hear a different opinion, but my stress level goes really high when I think about the fact that if I lose X drives, I lose all the data that I have ever owned. Having it spread in this way gives me more confidence that at least I won’t lose all the data at the same time.

As you can see, stress is my major driver :rofl:

EDIT: Forgot model numbers. This screenshot doesn’t include backup drives:

disk-models
disk-models1350×580 36 KB

  • MARS has 1 HDD green for backups
  • VENUS has 1 HDD green for backups
  • JUPITER has 1 HDD Exos enterprise for backups: ST14000NM001G

I also have a spare 8TB Exos enterprise HDD

6

With that many drive bays, I would consider:

  1. backing up your data to Jupiter
  2. purchase 3 more 4TB drives (for a total of 7)
  3. Create a new pool of 6 x 4TBin RAIDZ2 (should be a little under ~16tb usable).
  4. Then use the 7th drive as a hot spare. This gives you double your current storage, and you would need to lose 4 drives before you lost data (assuming you resilver with your hot spare before you 2 more drives).
  5. Then restore your backup of your data from Jupiter.

This should give you better read/write performance, and better redundancy. Also, 4TB disks are relatively cheap. Since you have a ton of drive bays you could make use of this.

Then consider purchasing 7 larger drives (8TB or larger) in a year or so and build another similar RAIDZ2 pool and you can slowly migrate off.

7

Could you expand a bit on the “better redundancy”, that sounds appealing but I’m not sure I get the math behind 1+mirror being worse than RAIDZ2 with 6 in terms of redundancy.

8

With RAIDZ2, you essentially have 2 parity drives. Meaning you would need to lose 3 drives to lose data. If you have drive + mirror, you could lose your data if you lose both drives. And if you have a “hot spare”, you can re-silver with the spare to replace the failed drive and this would mean you would need to lose a total of 4 drives to lose data.

I currently have 7 x 14TB in my server with 6 drives being in a RAIDZ2 and the 7th being my hot spare. I am remote from my server for about 6 months each year. This setup would allow me to lose 3 drives and I would still be functional and wouldn’t lose data.

Additionally, you get the read/write performance improvement as the data is spread across more disks.

9

How do you handle backups? There is no hard drive that’s 56TB.

I assume the probability of me losing 1 HDD + 1 mirror + 1 backup is extremely low. I do not have an offsite backup though, my drives are at my home (except family photos, those are offsite too)

10

It seems to me that you don’t seem to need more disk space at the moment, so (being parsimonious) I wouldn’t go out and buy any new drives.

But I would be looking for a better solution using your existing drives that is more automated and requires less administrative effort.

Firstly looking at performance requirements, you have a small amount of data for Jails (apps) and for this I would go out and spend a small amount on an SSD for a separate apps pool solely for holding the Jails and their active data (e.g. Plex/Jellyfin metadata). You can replicate the apps pool to a HDD pool as a backup. The rest of the data is not high performance, and you don’t need SSDs or mirrors, and RAIDZ would suffice. Media files will stream perfectly well from RAIDZ pools - you really don’t need mirrors for media files.

Then I would go back to first principles on backups, and think about what issues you would need to recover from - which I would classify as:

  1. Hard drive failure - use redundant disks.
  2. Server disaster - you definitely need a 2nd copy held elsewhere than the primary server - and ideally this would be off site (to handle e.g. a house fire). So you have c. 2TB of data on VENUS, and I would look to find a low-cost cloud storage solution (like pCloud) that you can sync to.
  3. Unintended modification or deletion - I would use snapshots to allow for recovery.

Then I would think about how I would build pools with the disks I already have if there were no data on them yet, and then try to work out how to achieve this by moving data within the disk space I already have without having to offload it somewhere else (and because the 14TB drives are relatively empty, this shouldn’t be difficult).

So, if it were me I would:

  1. Implement cloud backup for the data on VENUS
  2. Implement a 2nd SSD as an apps-pool, and move all the Jails data from MARS to it.
  3. Put all disks in your server i.e. the 3 existing backup disks and possibly the 8TB drive (though that could e.g. be kept for storing an occassional backup of your vital files in a fire safe or locally off-site).
  4. Move the data from MARS and EARTH to JUPITER and use all your 3TB drives to create a RAIDZ2 vDev / pool (called say MERCURY).
  5. Move the data from VENUS to MERCURY and junk the URANUS data, and convert all the 4TB drives to create another RAIDZ2 vDev / pool (called VENUS) - or potentially add it as a 2nd vDev to MERCURY.
  6. I would then look to move all the data off JUPITER and create a RAIDZ1 pool using the 3x 14TB drives (called JUPITER) which would hold all the media files. (For media files and only 3 HDDs you can probably risk RAIDZ1 despite them being big drives.)
  7. I would create datasets to separate the various classes of data.

This would leave you with double the amount of useable disk space (which would last you for the foreseeable future even with a snapshot backup strategy), with all drives redundant and fewer pools that you have to balance data across.

(I am not sure that this explanation is particularly well written, so please ask for clarification if it is unclear.)

EDIT: You also need to check whether any of the 3TB / 4TB drives are SMR, and if so they should only be used for non-redundant pools.

1 Like
11

Thank you, I’ll follow up later since I have an appointment now, however some things to keep in mind:

  1. My backup drives are not in the server, they are in a drawer and I plug them in to run the backups. It’s annoying because of the screws, but I don’t want another machine to host my backups and there is no cloud service I’m willing to pay for 22TB
  2. I’d rather lose partially my data than lose all of it at the same time, that’s a reason for the pool (nature of media files, if I lose part I have the others. For photos there is no replacement)
  3. If my machine breaks, I have no way to access any of that easily because I need another machine with at least X slots for the RAIDZ2
  4. I love having many slots, but all the machines with many slots seems to be very noisy (servers), if this one ever breaks, I’d love to buy a non-noisy one with many slots, but I suspect based on a cursory research that I wouldn’t be able to go further than 8. So I might end up with 8 slots instead of 24 at some point. I’d rather not, but if I have to buy another machine like this, I would definitely avoid another noisy server.
12

Note: none of my drives are SMR. I was very careful not to get those.
The backup drives are WD green so they cannot stay inside the server, I use them only for backups for that reason. There is some problem with noise and amount of drives in the same machine mentioned on the spec sheet, I think their max is 8 drives.

13
  1. I understand you don’t want to build another machine, but it might be the safest for your data. The 3-2-1 rule exist because there are ways for all of your drives to die together - bad PSU, power surge/lightning, fire, water damage etc.
    I know it would be an expense but you can use pretty much any old leftover PC and its worth looking into setting up a second machine at a friend or relative and send snapshots via VPN. And for 22TB you would only need a RAIDZ-1 with 3x10TB drives.
  2. Greater redundancy is usually better and you get that with RAIDZ-2 since you can have the extremely bad luck of the 2 drives that die be the mirror drives of a pool. Also during resilvering a drive is stressed to the max and more likely to fai and that js exactly what happens when you use RAIDZ-1 or mirror.
  3. There a few tower cases that fit 10,12,16 drives easily.
  4. What’s that bit about WD greens not being recommended to be >8 in a case? There are rubber grommets that reduce vibrations if the spec sheet mentions that as the problem.
14

Thank you, I really appreciate the follow up.

I’m not willing to put money into another machine, it’s a lot of money either way, I don’t have any other PC hanging around, they have all been repurposed already. The drives I use for backups are not connected to the server except for the time executing the backup (which is quite short, given the replication is partial). Yes it’s a risk, but quite limited.

Yeah in case of fire I lose everything but I don’t lose the most precious data, that is also backed up in the cloud (I need only 300GB for that), so it is 1 drive + 1 mirror of that drive + 1 data backup on a separate drive + copy in the cloud.
I’m ok accepting this risk, I lose my media, but if my house is on fire I have a very different problem, so as long as I keep my most precious data alive, I’m satisfied.

The fact that there is a tower that can fit 16 drives is amazing. Although it shouldn’t have just that, I really grew enjoying the “server” features this machine has: IPMI has become a lifesaver over time, do these towers have IPMI? I remember seeing some towers from Dell that are considered server machines.
Are they quiet? That’s an important thing that’s usually irrelevant for servers, but very relevant for a home server (I live in a condo).

  1. What’s that bit about WD greens not being recommended to be >8 in a case? There are rubber grommets that reduce vibrations if the spec sheet mentions that as the problem.

Cool. I remember reading on WD website that you shouldn’t use it with more than 8, the rubber grommets could definitely work, but I do feel uncomfortable using those vs seagate ironwolf given the whole SMR mess WD made.

Right now my biggest problem is that plugging the backup drives takes time: I literally have to put 4 screws on 4 drives to plug them in the server (it has those metal things that then get plugged in). I would make backups more often if I could speed up that process.

Just to clarify, I will potentially get another machine if I can shrink the size of my server (in physical terms), but just not now. I had very high expenses this year, my budget doesn’t allow me for anything else and finances have priority :slightly_smiling_face:

15

They can have whatever you like - every Supermicro and Asrock server motherboard has IPMI and they make a lot of mini and ATX boards.
Also nowadays PiKVM is a thing. There are cheap Chinese clones that are a plug and play unit.
Dell tower chassis are something I shy away from since it’s hard to fit anything not already there in their custom cases and they don’t support many drives.

As for silence Fractal Design R series have dampening layer on the inside and there was a similar Silverstone case, but I think it’s discontinued. Other that that you would have to do your own research. That said it depends on your drives too. The only thing that’s noticeable in my Fractal R5 is the Hitachi’s scrub, but that’s because those are 7200rpm and also click.

Then let me introduce you to drive shucking. But for the current problem you can just look for a busted drive on your local Ebay equivalent. You need only the pcb and the cables, and you have a 5Gbit external drive.

16

shucking seems to be a technique to buy hdds for cheap, but I already have the backup drives. I literally need a fast way to plug them into the server, instead of putting 4 screws each time and then removing it

17

Yes, that’s why I suggest you use the enclosures. You just plug the PCB from a shucked drive into one of yours and backup as you would to an external USB drive. You can even leave the PCB and its cables connected to your server and just swap your backup HDDs into it.

18

Isn’t doing anything over USB a very, very bad idea™ in TrueNAS?

19

Using USB for permanent storage is indeed a (very) bad idea.
Leaving a drive attached through USB to maintain redundancy during a drive replacement, because you’re out of SATA ports, is acceptable. (Not “permanent”.)
Using a USB-adapted drive for boot is acceptable. (Not “storage”.)

20

Personally I think this depends on whether you are looking at this from a data safety perspective or an availability perspective.

From a data safety perspective I have never lost data on by USB SSD boot drive - though I have come close a couple of times when the GPT Partition table was corrupted. But if I did, I have a backup copy of my configuration and of the other apps pool on the drive, and I can recreate the boot disk just fine if need be.

However from an availability perspective, using USB (or Thunderbolt?) ports for ZFS storage is not a great idea due to occasional disconnects.

If the USB storage is for e.g. an external stand-alone backup drive, then losing the connection means that any backup job in progress fails, but the rest of your NAS keeps working just fine.

If the USB drive is for primary storage, that drive will go offline, perhaps taking the pool offline, and the data won’t be available until you manually bring it back online, but TrueNAS will still be running.

But if the USB drive is your boot drive then by default operations to the drive will be paused until you manually bring the drive online again - which IME actually results in TN hanging and then you need to do a hard reset to bring it back up again. (This is a ZFS pool attribute. You can change it to fail the I/O instead - but IME this doesn’t stop the system from hanging. You can also change it to panic i.e. crash the O/S - and this is what I have now done, but for some reason it has not had an I/O fault since I made this change.)

IMO this means that USB SSD for a boot drive is an availability issue and should be avoided if at all possible (but if you accept this risk, then it works just fine).

IMO Debian needs some sort of automation to try to handle unexpected USB Mass Storage disconnects by trying to reconnect before reporting an error up the storage stack (in the same way that other protocols try to resolve temporary glitches without impacting the user.