Upstream data platform

Technology Reply created a Data Platform for an important Power & Energy client to collect data from different source systems in Real Time and make them available for Upstream analysis.

Background

The Data Platform aims to integrate different data types from heterogeneous source systems, each focused on a specific functional area. The main elements of Upstream functional areas are drilled wells. These are linked to the data gathered during the drilling operations:

  • Log, i.e. the information collected while extracting;

  • Rock and fluid samples from the subsoil, on which it is possible to perform laboratory analyses to obtain helpful information regarding the geological layers of the subsoil (for example, the physical properties description of the samples);

  • Subsoil drilling trajectories;

  • Stratigraphy, i.e. geological data determined by measures or analyses associated with deep ranges;

  • Seismic detection data;

  • Exploration map (in 2D and 3D) interesting for the client;

  • Territorial data related to the soil, sea, wind, etc.

Thanks to Oracle Golden Gate, which allows data replication in Change Data Capture mode, data transmission and its related acquisition bring a significant advantage to the platform and guarantee Real Time data availability to the final user.

Data are processed and standardized through ETL (extract, transform, load) processes after being collected from source systems in Real Time. During this process, data are aggregated, and eventually, if they satisfy Data Quality controls, they are inserted into the Business models realised following the users’ needs. A web application created with Oracle Application Express allows data visualization and data management on the front-end, which are central aspects of the Data Platform. The Web Application access has been implemented through authorization, authentication and a profiling mechanism to meet the client’s needs. After the login, users can browse, edit and manage data they can access (depending on the role) to create new technical and functional information. In addition, every user can see and operate only and exclusively on data of their functional areas. The main interface offers both an information tabular view with specific filters (related to each data type) and a geographical satellite map on which it is possible to perform visual searches. The platform becomes the single point of truth in multiple Upstream areas. It has to centralize such information and share it with other systems through an API Layer that can be queried from the outside.

Solution

Data are replicated in Real Time using the functionalities offered by Oracle Golden Gate to ensure a continuous data alignment between source systems and the Data Platform. The data load into the Data Platform is managed by Oracle Data Integrator, through which it is possible to apply transformations to the replicated data and insert them into the target Data Model.

Data from the ETL process can be aggregated and analyzed within APEX pages and published to external applications using Oracle API services. The different dashboards integrate Oracle Application Express Low Code components with custom components that fully satisfy the client’s requirements. The implementation of an orchestration tool using Oracle Business Process Management allows monitoring of all the processes started automatically or by Business users to ensure concurrency and consistency in the whole system.

To ensure the ETL processes’ correct execution, an Application Monitoring service has been implemented. This service can detect and correct possible data flow errors or infrastructure issues, and it has been customized by implementing a custom set of alerts sent by Oracle Enterprise Manager to promptly notify anomalies. Lastly, Business users can open tickets for anomalies or information requests on a DevOps platform, which can offer agile and immediate communication with the development team.

Advantages

The main advantages offered by the Data Platform are:

  • Real Time data replication through Change Data Capture

  • Different data sources integration into a single Data Platform

  • Data redundancy reduction

  • FE data visualization based on roles and authorizations

  • User operation centralisation

  • User’s data discovery facilitation