Ok, so another question regarding system backups and data transfer.
I’ve two systems at the m9ment. A heath-Robinson dell r410 with a patched together external jbod enclosure and a new replacement t620 power edge server.
Both are running scale.
On the old system I have a main datastore and a backup copy. I want to transfer the main (copy) to the new t620.
I’ve setup the new system with a new datastore and want to push all the data (6TB+) to the 13TB capacity of the new system.
Setting up an rsync job to copy the data runs for a while and then completes “successfully” at about 1.5TB.
So, obviously it’s not doing all the data. Do I just leave the replication to run every hour until it gets all the data to the new system eventually? Is there a way to force the system to do a complete (zfs?) block by block file copy?
Do I need to somehow setup some unix shares or ftp to ensure the data is completely replicated?
Struggling a bit here as the data store on the old server is split across the 4 internal hdds and some of the external array drives, otherwise I’d just stick them all in the new chassis and replicate from the old 4x2TB drives to 4 new 3TB ones….
Any help would be greatly appreciated. All I can find online is “do an rsync to copy a whole datastore”, but it’s not seemingly completing.
Rsync is file-based, and so it will not include ZFS snapshots (which are properties of ZFS), and depending on how it’s configured, it won’t traverse filesystem boundaries.
Yes, with replication, either from the GUI or the command-line in a tmux session.
You can get an overview of size usage (snapshot, clones, children, etc) with the following command:
So, what’s the best way to force a complete file transfer? Is there a command line option to ensure it’s fully replicated? I’ve definitely more files than the 1.5TB would be……
Looks like pushing from the old server rather than pulling from the new is still shunting data. Up to 2.45TB done now….
Though, just realised I could put the old drives in plus connect up the jbod and still have space to have the other 4 drives as a raid 5 group with enough spare space….
If the synch isn’t done properly tomorrow then I’ll export the old pool, port the drives across, slap in the hba and migrate the data that way… :).
Then I should also be able to pull the config and apps across as well.
Shame I didn’t think of doing that from the beginning…… :).
Got it up to about 4.7TB but the. It stopped and on restarting the job it won’t overwrite…
Rather than faff about with writing a new set of jobs etc, I’m going to take a spanner to it and swap the storage over….
If I backup the old config and migrate that across plus move all the old drives after exporting the datasets, it should just work right? :).
Plan is to pull the old storage over into the new box, replicate it to the bigger of the 4 disks then move them all back (possibly) to the old system, leaving the jobs connected on the new box.
Or, just keep the newer drives as spares, then look at putting a SAS jbod box together later (the current one is SATA).
An RSYNC job when setup from the Data Protection tab with the recursive option checked should backup everything under the dataset selected and run until completion. If that is not the case, RSYNC is setup incorrectly, the dataset being backed up is not what you think it is, or their is a failure in hardware. RSYNC is not the fastest and will take awhile depending upon the amount of data to transfer.
Replication between Truenase systems can be an option. It transfers snapshots and requires a snapshot setup to transfer. This is done during the replication setup. The first time a replication is run, all the files in the selected dataset will be transferred which may take awhile. After that only what has changed in the snapshot will be transferred.
In a bit of over simplification; You should decide on which method you want to use as you can’t have both an RSYNC of the data and a replication of the data in the same destination point on the destination server.
Currently I have the old drives ready to go in new caddies, just sorting out the networking (which stopped working completely) after porting over the old config.
Once I can login again, I’ll stick the drives in, boot up the jbod and try to replicate locally…
So, after a bit of faffing I pulled the 4 sata drives out of the old box and stuck them into the second set of bays (the first having the sas drives). I was worried about inadvertently mixing types but the full server health check showed no issues.
The storage pool was showing up as “unhealthy” on the old server and still are on the new setup.
So, replicating to the extra empty capacity and running a zfs scrub on the old disks.
I really want to move the data onto the sas drives but I’m not sure how to do that without having to reset up the shares and apps etc…
Old storage(1) - rsync to new disks(2) - remove old storage(1) - add more disks(3) - copy from (2) to (3) - setup (3) to run as (1) with all the vm’s, shares, rights etc……
Right, so struggling here with this as I think there is a snapshot issue I need to tidy up.
The original system storage pool is 12TB maximum size with 6.11TB used.
The new pool is 7.8TB in size.
So….
The new pool is filling up completely even though from a capacity standpoint there should be enough space to hold the data.
So, is the replication trying to do an exact mirror (size and all)?
Is it trying to copy across all the snapshots and filling up the space?
Is there a way to replicate the pool that JUST copies the data and nothing else? I’ve read the guides on here so I can swap over the two pools and ditch the problematic old drives.
I could possibly add two more 2TB Sas drives into the jbod on their own channel but that’d still not be enough.
Anyone got a solution? Team numpty needs a plan here :).
So, tried specifying a new naming convention and it still filled the drive.
Funnily though, everytime I ran the jobs previously it would say something like “syncronising 1 of 200” or similar large numbers so got the feeling that it was trying to replicate large numbers of snapshots for some reason. Without knowing they were incremental ones, I guess it would be writing all the data again and again?
Previously I had run a manual snapshot then setup the replication to copy the info across.
However, just set up a new replication with a custom name, telling it to create a snapshot if needed.
Started running it and it does say “synchronising 1 of 1”, so maybe this has nailed it?
