File Migration Series – Overview

Hello,

As you may remember, I did exclaim lately I bought new storage.  I am moving from an older QNAP (TS-459 Pro II) and EMC Iomega (PX4-300d)  to a new Synology 1813 and have been talking with people lately about moving to new storage so I thought I would blog my experience of the move.  This series is about moving to new storage and what you need to be aware of.  While I do not have the new storage yet, I am going to start the series today.  This will be, I trust, good info to share.  Copying files to new storage is not as easy nor as simple as it sounds.  You need to preserve important information about the file as well as the file it self.  The series will look something like this:

  • Overview – this article!
  • Planning – Diagramming  / testing copy speed
  • Designing the migration plan
  • Migration – doing the actual migration
  • Updates – Login scripts and files
  • Testing – did it really work?  Are we good to go?
  • Finish – what did we learn

Overview

Moving a quantity of files is both simple and hard.  It is simple as you do not even need to look for and find Robocopy any longer as it is part of the Operating System in newer versions.  But it is still hard as we do not know how much time we need, or if the copy tool preserves all the file information.  So here is a list of things to think about.

  • You should plan out the process, and I often like to have a diagram for reference.
  • You do not know how long the copy will take, but likely it will take a very much lot longer than expected.
  • You do not know how many other file processes are ongoing that may impact  you – files already open are harder to copy – i.e. Backup running at same time for example.  So you need to test so you can figure out how long this will take.  In the bunch of times I did this professionally for customers it often required 1 or 2 weeks of nightly copies.  I only knew that after I tested and sometimes only by looking in the copy logs to see why things took unexpectedly longer.
  • Think where things are – try for the shortest path when you do the copy.
  • You will need an outage for the cut-over – not just for a ‘catch up’ on files, but you will likely need to tweak things like login scripts – or Excel spreadsheets!
  • Sometimes you might need to use multiple copy machines but that is very dangerous and easy to screen up.  I recommend you avoid that if possible.  But if you need them use them plan carefully – it is sometimes worth thinking more threads vs. more machines.
  • Everyone uses Robocopy but I have found it less than ideal and I switched to using Secure Copy.  Faster and more powerful.  This experience is old, and so it was good to see that Secure Copy is still around – but I am astounded at the cost of it.  I may not use it after all!
  • Understand the files you are copying as if they are mostly little files, or mostly big files will impact the copy. Again, the test should help with this.
  • The post migration test to confirm everything copied right is very tough.  You need to test a lot of things – not just permissions / owners of files and folders, but if there is Excel spreadsheets in your migration you will need to worry about the likely situation where there is paths in those spreadsheets that need to be updated!

So this is a good list of things to think about.  What will see see next?  I need to diagram out the structure of what I am moving – I have two arrays to consolidate into one.  I have quite a large CIFS implementation with both files and file information I don’t want to lose.  I also need to test how fast things might go.  So you will see most of this next.

Michael

=== END ===

Tagged with: , ,
Posted in Home Lab, How To

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: