Managing Expectations for Data Migration

The workflow of a project is marked by the path through which data flows from team to team. Each time data is touched it must also be validated against itself for relational integrity and against other source data for dependencies. These validation steps must occur before the data is passed downstream to the next team and, as the project progresses from early testing phases toward go-live the validation must occur faster and with more accuracy.

It is common practice to hold data for manual manipulation until the last possible second or beyond before passing it along to the data conversion team. When time runs short data may be passed along without first being validated. Teams can develop expectations over time with regards to how much of the unfinished work a data conversion team is willing to do. The data conversion team lead can mitigate the damage caused by data that is delivered behind schedule or in poor condition by managing these expectations. Some rules and guidelines that have been used in the past to successfully manage expectations are:

Refuse to load data out of order.

SAP objects that have dependencies cannot be loaded before those dependencies are first loaded and dependency data often comes from different teams. Completing a data load by loading dummy dependency data in order to appease a too-eager-to-test user will set a dangerous precedent and fills the test system with dummy data. A system full of dummy data complicates other tasks or creates new cleansing tasks. Additionally, the expectation going forward will be that any data, regardless of the status of its dependencies, will be loaded on request regardless of how much additional work is created.

It may be enough to explain to the user that the data requested cannot be loaded until its dependencies are loaded. If the pressure to load data with dependencies persists then direct the user to the team responsible for producing the dependency data.

Refuse to rush data loads during early testing phases.

In an SAP implementation many people will have idle time while waiting for a process to complete and the temptation to expedite those processes can be very strong. A user who is waiting to test loaded data may complain to a manager that his time is being wasted and undue pressure is then applied to the teams performing the extract, transform and load. Rushing these processes not only increases the likelihood that the data will be incorrect in the test system, but sets the expectation of how long a step takes.

If a master data champion believes that a load can be performed in just two days then he will wait until the Thursday before go-live weekend to deliver the transformed data. One week is a very realistic minimum for the loading of master data objects before go-live weekend.

Handle data delivered at the end of a business day at the beginning of the next business day.

A common tactic employed by users who work on data with a loading deadline is to wait until the end of the business day and then hand off the data to the next person responsible on the way out the door. This allows the user to avoid being further involved and still claim that the data was technically delivered on-time.

The temptation to work late into the night can be very strong, but compensating the late delivery of the data by working extra hours creates the expectation that data delivered at the end of the business day will still be loaded by the deadline. Allowing the data to wait until the next day sets a precedent and makes it clear that data must be delivered before the end of the day on which it must be loaded.

Maintain daily backups of source data

Another common tactic employed by users who run short on time is to update source data after it has been loaded into a test system. Some users actually believe that modified source data is automatically updated in an SAP system. Other users will ask why the data in the test system does not match the source data even when the obvious answer is that the data was changed after the load.

The expectation that data in a test system will automatically be changed by modifying a source file after the data has been loaded is a very common one and is one of the most important expectations to manage. A great way to demonstrate that this does not occur automatically is by keeping a copy of the source data in a backup location that users cannot access.

Some users will go so far as to change source data after it has been loaded into a production system and then blame the data conversion team for loading an outdated source file or for making a mistake. If the source files are managed in a centralized location without revision tracking then it can appear that the source data was loaded incorrectly and a reliable backup is the only way to demonstrate that the data was loaded correctly.

Explain that data must be validated before it is ready to load.

SAP performs validations on data that is manually loaded and many users expect that data need not be validated because SAP will catch any errors. SAP will catch inconsistencies, missing dependencies and configuration issues during an automated load process, but SAP will not identify problems with the source data. The professionals performing the data loads must identify the errors in the source data, develop tools to catch the errors in the source data, generate error reports, explain how the source data must be corrected and then wait for the corrected data to be delivered.

After the corrected data arrives the team must develop methods to load only the records that failed previously in order to prevent the creation of duplicate records and/or more errors and then attempt to load again. If SAP catches more errors then the process begins again.

Relying on the team performing the data loads to validate his data is another tactic employed by the same user who hands off deadline data on the way out the door. See the flowchart below to better understand what happens when data is not validated before being delivered to the load team.

Explain that failed data records will not be reloaded, but tested again during the next testing phase.

Users who rely on the data conversion team to validate data will, after receiving an error report, modify the source file and send it again with the expectation that the team performing the loads will develop some method whereby only the failed records will be re-tested. By allowing those failed records to remain failed, the expectation on the part of the user is that the data must be validated before it is handed off to the data conversion team.

Allowing failed records to wait until the next testing phase highlights the importance of validated data. The validity, integrity and consistency of source data is not very important to the user if the data conversion team lead will allow for the testing and re-testing of the source data in increasingly complex and time-consuming tasks.