Data Migration – Does it Need to be Scary?

Data migration sm

In the IT world there aren’t many phrases that can put a damper on a conversation as abruptly or as effectively as “Data Migration”.

Those two apparently innocuous words all too often cause even the most hardened and seasoned of IT professionals to break out in a sweat and rapidly circle around the subject to something – anything - else.

Or, perhaps even more unhelpfully for those that have never been involved in a Data Migration before, a colleague may pull a grimaced expression on hearing those words, then swiftly exit stage left with a pat on the back and mysterious, ominous “good luck!” – leaving them even more in the dark and with an all-new concern that something not that pleasant may lie ahead. But why is this the case and is this fear of Data Migration justified?

Check this infographic that summarises 4 key steps to tackle data migration challenges.

Data Migration is often scary because it is a task that often catches people out. Many can see it simply as a case of moving data from A to B, which essentially it is. However, the amount of preparation that is required to achieve a smooth migration is often underestimated and data is not very forgiving. There isn’t much room for error – the migrated data is either correct or it isn’t. The ever-increasing volumes of data that needs to be migrated isn’t making the task any easier and the regulatory responsibilities placed on data administrators, by the likes of GDPR, only add to the pressure. The migrated data must be correct.

An added challenge for those tasked with carrying out the migration is that there are often many unknowns. The data may appear one way in the user interface but behind the scenes, in the underlying database from which the migration will read the data, the data can be stored, and even named, completely differently.

In the UI an “Account” form may suggest the data is all from one underlying table, but the reality may be that it is in fact a composite, from many different tables. And a field with a friendly UI display name, such as “Territory” may have a database schema name that is totally different and obscure, such as “UD_Text1”. So, one of the first tasks is to identify which fields from the UI need to be migrated and to then ascertain and document where these are located in the database. Often clients are not in a position or experienced enough to do this, but the system may be bespoke, old or both – and it can be a challenge for an IT Partner to navigate and reverse engineer – certainly alone.

The analysis must be a team effort, with the Client, as the Data Owner, responsible for clarifying what the data is, what needs to be migrated and how it links together in the UI - with the IT Partner helping them, if required, to translate that to the database.

The “Four Pillars” of Data Migration

Due to the many variations in source and target system - and the types and quantities of data to be migrated – each Data Migration can feel like a first, which in turn can contribute to the fear and wariness mentioned above.

But rather than focusing on the differences, it is more constructive to focus on the constants – what factors or steps will always be the same? Taking this approach can lead to quick progress and can help break scaling the mountain down into manageable tasks.

Data migration pillars

1. Access The Data

A seemingly obvious requirement, which often causes delay, is that analysing and migrating data requires access to it. Whether the data is in a spreadsheet, on-premise database or cloud-based system, IT Partners will need read-only access to this – which can take time to provision the appropriate credentials, VPN access etc.

2. What Data to Take

As the Data Owner, only the Client can determine what data is needed for their business, what data can legitimately and legally be retained, and what data, if any, can safely be left behind in the legacy system. The earlier internal conversations can be had to try and prepare answers for these questions, the better.

Also, striving for a “minimum viable” dataset nearly always yields the most successful migrations. It is too easy to say “take it all” – often through fear of leaving anything useful behind – but a Data Migration is the prime opportunity to “clean house” and populate the target system with only the best and most useful data that follows the principles of the business’ Data Retention Policies. What’s that – you haven’t formalised any, yet? Again, this is the perfect opportunity to do so – not only to ensure the target system is populated appropriately but also so that measures can be put in place to keep the data compliant.

3. Where To Find It

Having determined what data is required the next task is to identify where to find it. This could be columns in a spreadsheet or fields on the source system. For the latter, initially identify and document where the required data can be located in the front-end – then assistance may be required to map out where the data is behind the scenes, in the underlying database.

4. Where To Put It

The final piece of the puzzle is to identify, in the target system, where all the data needs to go. Most data might be migrated 1:1 but some entities may need to be split or merged, some option-set values changed or re-mapped and possibly some data may need transforming or cleaned en-route – for example, capitalising the first letter of name fields or removing any text from phone numbers.

With these four elements completed, it should be possible to build a “Mapping Schema” that identifies exactly what data needs to be migrated – where from, where it needs to be put and if and how it needs to be transformed. This is the recipe book for the Data Migration and the key to a successful process that does not need to be feared!

How to Tackle Data Migration Challenges: Infographic

Data migration infographic Craft Upload

Thinking about a Data Migration?

If you’re about to embark on a Data Migration, or are considering an integration with Dynamics 365, Dataverse and the Microsoft Power Platform, contact us to find out how Preact can help.

Keep up to date! Subscribe to our blog

Join our mailing list to receive our emails about what's new in Microsoft Dynamics 365 and the Power Platform.

Subscribe now