Skip to main content

Celonis Product Documentation

SAP Real Time Extension: Solution Overview

Overview

Data Extraction from SAP is the first step in the process analytics pipeline. The data is replicated from the SAP database into Celonis Analytics Storage, where it is converted into an Event Log and loaded into the Process Mining Engine.

In an ideal world there would be a real-time integration between SAP and Celonis, so that updates in transactional ERP data are immediately propagated into the Celonis Event Log. However, in reality this is not that easy to do, and one of the main challenges is to identify the updated records, so that the load is isolated only to them. One way to overcome this is to use timestamp based delta extractions. The idea is that each record has a timestamp field which is updated each time the record is modified. So by filtering on that timestamp we can isolate only the records that have been updated after the last load. However, this approach does not work in all cases, because some very large tables simply do not have timestamps, making it impossible to identify the updated records. As a result, a full load is carried out for these tables, loading the whole data during each run. In addition to that, the Delta Load may be very slow sometimes, because the filtering is done on non-indexed fields.

To overcome this limitation and get one step closer to real-time data extraction Celonis has come up with a new approach on how to extract delta. The basic idea is that the relevant tables are continuously monitored for changes, and once a record is updated, its id is logged in a dedicated table. The next time an extraction is run, only the records that have been updated and logged will be extracted. To be more specific, the extractor will inner join the tables and get only the updated records. Since the join will be done on primary records, the delta filter will be very effective performance wise.

Technology Details
Set Up

As part of enabling real time extractions a change log table and a trigger should be created for each source table that should be extracted in real time mode. The trigger listens for create, update, and delete events on the source table, capturing primary keys of the modified records, and inserting records into the respective change log table The change log tables and triggers are created during the setup. It is important to note that even though the setup is done from SAP GUI, the triggers are indeed created on the database level via a native SQL statement in the syntax of your underlying database.

The triggers and log tables are created and maintained via transaction code /CELONIS/CLMAN_UI which becomes available after the real time extractor is imported into SAP.

41191829.png

The triggers are not SAP DDIC objects, so you can create/delete them manually in PRD environment without unlocking the SAP client for changes.

The log tables, however, are SAP DDIC objects, so they are usually created in DEV and transported forward to the QA and PRD.

Extraction Logic

The real time extractor works with the standard Celonis continuous SAP extractor (you can find more info on the continuous extractor here). It can be turned on/off anytime from Celonis EMS.

In case the real time extraction mode is enabled, the extractor service calls the RFC function /CELONIS/EX_CL_NEW. For each table, it checks whether real time extractions have been enabled or not. If so, modifies the extraction filter to ensure that only the changed rows are extracted, and then delegates the task to the /CELONIS/EXTRACTION program which follows the standard extraction flow, by writing the data to csv files and sending it to the extractor service and then to EMS.

Once the extraction is complete the RFC function /CELONIS/EX_CL_SET_EXTRACTED is invoked to flag the extracted records so that subsequent extractions only receive new changes. If the extraction failed, then the function is not invoked, records are not marked as extracted, and subsequent extractions will receive the same (and all new) changes until successful.

The digram below explains how the real-time extractor operates.

22122343.png