File Migration Series – Design and Testing

Hi everyone,

We are continuing today with our File Migration Series, and we are looking at design and test in this article.  You can find the series, overview, and planning articles if you want to see the other parts of this series.

What we will do today is do a test copy.  Meaning we will select a subset of data, and record some of the metadata and do a copy and see if it works.  While I am sure it will, I want to be able to confirm my tool will copy both files and metadata and see if there are any errors.  Which normally there is, and get an idea of how long things will take.


We need to do a bunch of things before the test starts.

  1. We need to copy a subset of data.  I have 2 or 3 folders that I will copy and it is around 65 GB.
  2. We need to prepare by knowing what the owner / permissions are for a number of the files.  Maybe even tweak it a little.
  3. What command line for our copy tool should we use?
  4. We need to have a test document so that we know what to check after the copy.

Here is my Evernote held actual setup and test doc.


So I have done the setup part.  Note the deleted user setup item?  That is to learn how a missing owner will be dealt with.  A few of my customers have started to not delete accounts for users who leave but rather disable them, and at some point in the future (read never) they will deal with it nicely.  But there are still more customers that delete accounts so I am curious how this will be dealt with.

I have a Robocopy command line I think is the best to start with (command line below is one line).

robocopy z:\ y:\ /Copy:DATSO /MT:12 /R:5 /V /E /TEE /w:5 /log:test.log

After I check the time length for the copy, and any errors, I will perhaps tweak the command line.  While I am using robocopy, if I could I would be using SecureCopy – no budget to get it or a NFR.  There are other tools that I will include a link to below but Robocopy is easy and available. Note: if you care about the folder dates / times use the /dcopy:T command with the command line above.

Start Time

OK, lets give this a try.  Z: has the files, and Y: is empty.  I am on a VM that is Win7 and has access to both file shares (note: I am using the robocopy.exe that is in Win7).  Both file shares, and the VM are on the same network with no routers between them.


So I hit return.  And shortly later I get this.


The error above is because of the folder with the deleted owner.  It is in my test to check who the owner is now.  With Robocopy it will be the account of the person who is running Robocopy.  That is not always good but that is what it is.  SecureCopy will allow you to set a default owner that will be used in cases like this.

When we finish we see the following screen.


It looks like our copy went pretty well.  It took about 16 minutes or so to copy and I see no errors.

Test Time

We now need to do a couple of things.  We need to go through the log carefully and see if there is any issues.  We also need to use our test document to check the files and folders to make sure they have the same AND correct permissions / owners.

We need to confirm that the specific owner and permissions we checked on the source match what is on the destination.  For the file who owner we deleted it will likely have either a SID or the operators account – in this case mine.

If the owners, and permissions match we are happy indeed.  But they do not always match and that is why we are testing.  If they don’t match often it is due to things like Auditing being turned on the source, deleted owners, or extended attributes in use.  With Robocopy we do not have many tools to deal with these sorts of issues but again SecureCopy does.  So if you have some of these issues that Robocopy doesn’t deal with you will need to change tools.

We also learned we could move approximately 65 GB in 16 or 17 minutes.  Not bad.  In my lab I also learned that I can likely use Robocopy as my owners, and permissions were retained.  They did not look quite the same but close.  My source had Enterprise Admins listed in the ACL but not on the destination for some reason.  If the file copy time is not good, you can try redoing this test but increment the /MT:12 to a higher number.  Try doing an increase of only 2 at a time.  It will go slower at some point.

You can also do things faster by having two Robocopy stations doing copies.  But you will need to be very careful at designing what they copy and making sure it works as you expect.  I would suggest not doing this and using a tool such as SecureCopy – it does have more features that are very useful, but it is also – in my past testing, much faster.

What did we learn?

We have got a good command line now, and we know how things are going to play out.  We know that we likely should use several days of copy time so that we don’t miss anything.  We know to check the log each day as well.  If I had many TB of data I could figure out how long it would take to copy.  Depending on what kind of file shares you are copying you may have a lot of open files.  That is another reason to do multiple copy days and check the logs to make sure you do eventually get the open files copied.  Some of the other tools do much better then Robocopy at dealing with open files.

Useful Links

Robocopy overview

Robocopy command line info

Richcopy – not recommended as it is discontinued and has no support.

SecureCopy product info, recommended

Syncback – product info, a good possibility

File Migration Series

Whats Next?

In our next article we will look at doing the actual migration.  See you next time!


=== END ===

Leave a Reply