Planning and surviving a data migration

Data migration sounds straightforward. After all, it's simply moving your data from one location to another. But in reality, a successful migration needs careful planning and strategy, or it can take longer to execute, as well as ending up much more expensive than you had hoped for.

In our recent webinar, our Vice President of Product, Branislav Repček, provided an overview of the different components of data migration, the stages involved, and how to ensure your data migration is a success. This post summarizes some of the key points, or you can watch the whole video below:

Data migration: 3 components

Before you begin with your data migration, you need to first familiarize yourself with its three key components:

Your legacy systems. This can be one or multiple systems. They're often heavily customized and business-critical, so it's crucial to get the migration right.
The data management system. This is the middleware you use to get your data from point A to point B.
Your new system. This is the target system to which you're migrating, most likely for increased speed and agility, improved costs, and better data integration and accessibility. These days, this usually means moving to the cloud.

Now you know the key components, it's time to move on to the next phase in your project.

Data migration: 4 stages

There are four stages in the implementation process of your data migration. While it can be tempting to rush through the first three stages to save time, this will only create problems during the execution.

1: Strategy

During the first stage, you should identify your migration scope, which will allow you to budget for the project in advance. It's important to realistically represent this as underestimating could bring you in over budget down the line, or even make the migration seem unnecessary.

Once you've figured out the scope, you can choose how to migrate your data:

Big bang migration vs trickle migration

In big bang migration, all the data gets migrated at once, meaning the project is completed within a relatively short period of time.

With trickle migration, however, you split your data into smaller batches to migrate. This usually means the migration takes longer but is easier to control. It's also easier to verify the success of your project.

As this process means you have to run your legacy system and new system simultaneously, it's more expensive than big bang. But it's worth considering if you have a large volume of data or the data is complicated.

big bang vs trickle migration

2. Planning

Now that you have your strategy, it's time to plan how to put it into action. Here are some steps that customers should take:

Identify which data to process first. The more complex the data, the more likely it is to cause problems. That's why it's important to migrate complex data first so any issues can be mitigated early on in the process.

Outline best-case scenario and worst-case scenario. By knowing what to expect from a successful migration, you'll be able to see if the execution is going smoothly. And a worst-case scenario will allow you to more quickly implement your contingency plan should anything go wrong. This brings us to the next step.
Make a contingency plan. This will differ depending on what kind of migration process you've chosen. But having a contingency plan in place will make it easier to deal with any problems and roll back your migration if necessary, whether it's small sets of data or the whole lot.
Back up your data. It's possible that the data from your legacy system could break the data that already exists in the new system. It's also crucial to test your backups so that time isn't wasted during restoration.
Have realistic timelines. An unrealistic deadline will affect how much time you have left for your contingency plan and backup restoration. It's better to allow for all possible issues and pleasantly surprise your stakeholders with early execution should no problems arise.

3. Technical analysis and testing

This stage will allow you to take an in-depth look at the kinds of data that need to be migrated, as well as the technical skills your staff might need during the process. During this stage, you should also identify what data needs to be cleansed before the execution.

Identify the quality and sensitivity of data. The complexity and sensitivity of your data should dictate the order in which it is migrated and how it is handled. Regulations like GDPR should be taken into consideration at this point. You should also identify any problems with the data and whether it needs to be cleansed.
Decide if training is necessary. If you are relying on your employees for any part of your migration process, this could be the time to provide some technical training.
Data cleansing/ management. Cleansing corrupt and incorrect data before the execution stage will save you time and effort down the line, as well as making sure the data in your new system works. It will also allow you to ensure that the data is reliable and trustworthy for your users.

4. Execution

The final stage in your migration should be more straightforward, thanks to the first three stages. But it's also at this point that something can go really wrong. Here's how to ensure your migration runs more smoothly:

Communication. Stay in communication with the relevant parties and give detailed updates. Contact them as soon as you know of any setbacks, like your migration taking longer than expected. This is especially important for trickle migrations that can run for a relatively long time.
Keeping time aside. Providing reports and updates will take time, so plan ahead for this and reserve some slots in your timeline for just that.
Hypercare. This is additional help that might be required right after the migration is completed. It can be necessary when the people using the new system don't understand it, when something hasn't been configured properly, or if there are issues with your data.
Automation. Manually executing your data migration will be time-consuming, expensive, and prone to human error. You should automate every possible step of the process. This will make it faster and more accurate. It will also allow you to run better, repeatable tests. And it will also cost you less, as long as you've found the right partner to help.

Plan your data migration with CloverDX

Data migration takes a lot of planning to get right, and even then there can be issues that will make the process costly and inefficient.

With CloverDX you can:

Connect to any source and target systems on-premise, in cloud, or both.
Handle messy or complex data by building complex transformation logic into your data process.
Automate processes, making it easier to iterate and re-run, reducing time to delivery and expense.
Stay up to date with transparent reporting at every stage of the migration journey.

And if you need extra resources or expertise when time is tight, our services team can help.

For more information, watch the full webinar on Planning and Surviving a Data Migration. And if you'd like to chat to us about your data migration plan, just drop us a line.