Data migration is the process of transferring data from a source system to a target system. It is a core task for any data storage professional.
Data migration is important because it is a necessary component to upgrading or consolidating server and storage hardware, or adding data-intensive applications like databases, data warehouses, and data lakes, and large-scale virtualization projects. Data migration may also occur within systems built on HDD or SDD, or between in-house systems and cloud storage.
Types of Data Migrations
All data migrations are not cut from the same cloth. The usual migration suspects include storage, database, application, cloud, and business process migration.
- Migrating Storage. IT migrates data during a storage technology refresh. The goals of technology refreshes are faster performance and dynamic scaling along with improved data management features.
- Migrating Databases. Migrating a database can mean moving between platforms, such as on-premise to the cloud, or migrating the data from one databases into a new one.
- Migrating Applications. Application migration can mean moving data within an application, such as shifting from on-premises MS Office to Office 365 in the cloud. It can also mean replacing one application with a different one, such as moving from one accounting software to a new accounting platform from a different vendor.
- Migrating to the Cloud. Cloud migration moves data from on-premises to a cloud, or from one cloud to another. This type of data movement is not the same as backing up to the cloud: data migration is a distinct project that moves data from the source environment to populate the new one.
Note that data migration is not the same thing as data conversion or data integration. To clarify:
- Data Migration. Moving data between storage devices, locations, or systems. Includes subsets like quality assurance, cleansing, validation, and profiling.
- Data Conversion. Transforms data from a legacy application to an updated or new application. The process is ETL: extract, transform, load.
- Data Integration. Combines stored data residing in different systems to create a unified view and global analytics.
While data migration at its simplest is the transfer of data, the full context of data migration includes several component processes.
Data Migration Challenges and Risks
Data migration has the reputation of being risky and difficult. It’s certainly not an easy process. It is time-consuming with many planning and implementation steps, and there is always some risk involved in projects of this magnitude.
Data Loss
During the data migration process, data loss can occur. On a small scale, this may not be a problem – no one may ever miss the data, or IT can restore files with backup.
However, catastrophic data loss is different. In the case of a short-term connection failure, IT may not even know that the short-lived failure abruptly terminated the migration process. The missing data goes unnoticed until a user or application calls for it – and it’s not there.
Compatibility issues
There are also compatibility issues in data transfer, such as changed operating systems and unexpected file formats; or confusion over user access rights between the source and target systems. Although the data is not formally lost, the business cannot access it in the target system.
Poor execution impacts the business
Many IT departments decide to do a migration project in-house to save money, or the management team decides it for them. But do-it-yourself data migration is rarely a good strategy. Migration is a risky business with major business implications, and requires extensive expert attention.
A poorly run data migration project causes extended downtime, loses data, overruns deadlines, exceeds budgets, and results in sub-par performance.
The Process: How to Succeed with a Data Migration Strategy
Despite the difficulty and risks, IT can ensure a successful project within budgets and deadlines. It takes expertise, strategic planning, management buy-in, and software tools.
A working data migration plan will include the following:
Budget for Expert Help
Many IT organizations prefer to be hands-on, and some migration budgets do not allow for expert advice. However, unless IT already has migration specialists on staff, they will save money and time by hiring consultants who are experts at data migrations.
Plan the Strategy
Understand the design requirements for migrated data including migration schedules and priorities, backup and replication settings, capacity planning, and prioritizing by data value. This is also the stage where IT decides on the type of migration implementation schedule, sometimes referred to as “big bang or trickle.” Let’s look at these terms.
Big Bang migration completes the full transfer within a limited time window. There is some downtime during data processing and movement, but the project is completed quickly.
Trickle migration carries out the project in phases, including running source and target systems in parallel. Trickle migration is are more complex than Big Bang and takes longer, but has less downtime and more testing opportunities.
Work with Your End Users
Treat the data migration project as a business process instead of simply a set of technical steps, and involve your end-users. They will have understandable anxiety over the success of the migration project.
Work with them: understand the data rules and definitions, what data is subject to compliance, and priority data that should migrate first. Also understand what they are hoping for from the move: Analytics? Better performance? An easier way to issue legal holds?
By taking the time to work with end-users, you will have a more successful data migration project in less time and at lower cost.
Audit the Data and Fix any Issue
Know how many TBs of data you are migrating, and target storage capacity and growth expectations. Database migrations require auditing the source database for unused fields, obsolete records, and database logic; and making changes before migrating data to the new platform.
Migrating storage is easier because you do not have to update the older storage and map to the new. But migrating data between storage systems is not as easy as simply copying data from one secondary system to another. Use software tools to locate dark data, and delete or archive them properly before the migration.
Delete obsolete files, abandoned e-mail accounts, and outdated user accounts. Dedupe and compress the source data if you are moving data over the WAN, then migrate and test.
Backup the Source Data Before you Move It
If the worst happens and you lose data during the migration, be prepared to restore it to original systems before trying again. Best practice is to create backup images that you can immediately restore to the original system should the migration lose data.
Move and Validate the Data
Invest in automated data migration software that allows you to schedule staggered migrations of data subsets, validates data integrity in the target system, and issues reports for troubleshooting and verification. Protect databases during active migrations with a software tool that syncs the source and target databases in real-time.
Final Test and Shutdown
Once you have migrated all data, test the migration using a mirror of the production environment. When it all checks out, carefully go-live and conduct final tests. Once the new environment is running smoothly, shut down the legacy system.
Then make it easier on yourself for the next data migration, because there will be one. Instead of spending expensive resources to update the source data before the move, institute governance controls and analytics in the new environment. Continuously monitor the migrated data for orphaned work sets, unusual access patterns, and security. The migrated data will perform better in its new platform, and the next data migration will go faster and smoother.
