Data automation flows migration & data quality framework setup

SUMMARY

Avantage Reply assisted a large European bank to migrate data automation flows on Dataiku, a modern data platform. It has allowed the bank to disengage from its former platform which has become too expensive and less efficient.

After completion of this task, data quality controls were set up by Avantage to improve the accuracy of the data and related reports. The aim of this task was to align the bank’s data governance with the IFRS9 standard requirements and ensure the quality, reliability and completeness of the data as required by the BCBS239 regulation.

CUSTOMER GOALS

The main purpose of this project was to move the client’s analysis and reporting process to Dataiku and show all the possibilities given by the platform so that users can switch on it easily.

The second goal was to improve the data quality for the sources used in applying the controls.

To achieve them, several tasks were defined, which included:

Data flow migration

• Migrate all data automation processes on Dataiku using Python scripts.
• Facilitate and/or improve the efficiency of these processes with automation when possible.
• Teach users how to use this new platform and specifically how to use the new automation programs.

Data quality

• Set a list of controls for all databases in service by users.
• Build a program that evaluates all controls and returns data quality issues for each one.
• Develop a program to check the consistency of the data from multiple sources.

Finally, thorough documentation was produced with project definitions and tutorials.

CHALLENGES

Several Challenges were encountered during this project:

1. The programs on the old platform were not written by IT developers. Sorting was necessary between parts which were outdated and others still in use.
2. Lack of specifications for some programs, that we had to create from scratch. Additionally, users were too busy with production tasks to answer all the questions raised.

SOLUTIONS

In order to meet the client’s expectations while managing the challenges, we have followed the principles below:

• Analyse the outputs from the existing programs to establish a list of all fields and indicators to compute.
• Observe the method used to compute each field and see if it can be optimised or simply encoded in Python.
• Check-ups with the users at key steps of the projects allowed them to follow the changes and validate our hypothesis.
• Perform as many as possible non-regression tests at key steps of the data flows.