RFC module overview
The entry point of the extraction process is the RFC enabled function /CELONIS/FM_NEW_EXTRACT. The extractor service passes several parameters to this function, i.e. the table, columns, filters, and other extraction metadata. The /CELONIS/FM_NEW_EXTRACT then creates a background Job, scheduling it to run the program /CELONIS/RP_BG_EXTRACT.
During an extraction, a cursor on the table that is being extracted is opened with a default buffer size of 10,000 lines (the number of lines held in memory at a time). Those lines are written to CSV files in the location specified by the FILE path Z_CELONIS_TARGET. When the CSV file exceeds a specific number of lines (default size: 50,000 lines) it is closed and then - depending on the specified compression type - compressed. Afterwards the extraction continues writing the next file.
The extractor actively polls the directory Z_CELONIS_TARGET via the RFC function /CELONIS/FM_GET_EXTR_FILE_LIST. As soon as it identifies a CSV file in the directory, the function /CELONIS/FM_GET_EXTRACT_FILE is invoked to fetch the file from the SAP file system to the extractor server. Once the transfer is complete, the file is removed from the SAP file system via the RFC call to /CELONIS/FM_CLEANUP_FILES. If the user cancels the job, the extractor calls /CELONIS/FM_CANCEL_EXTRACT and cancels the Job in SAP.
While the extraction is in progress, the extractor actively polls the background Job status via calling /CELONIS/FM_GET_JOB_STATUS. Once the Job is completed, the extractor fetches the job logs via the /CELONIS/FM_GET_JOB_LOG and publishes it to Celonis as a part of the extraction logs.
Filters
Filters are appended on the select statement of the cursor. It is important to note that arbitrary filters are possible on the tables meaning that the performance of an extraction can be influenced greatly by correct filtering on indexed columns.
Joins
Joins are not translated directly to the select query but instead each join will use its own cursor. After a chunk of the join partner has been read we use a FOR ALL ENTRIES statement for selecting from the target table. Additional filters are appended as usual.
Other RFC operations
Selecting tables/columns to extract: The user can select which tables and columns to extract in Celonis. The table list is displayed to the user via calling the functions /CELONIS/FM_GET_TABLE_LIST and /CELONIS/FM_GET_CHECKED_TABLES. To list the columns the function /CELONIS/FM_GET_TABLE_METADATA is called.
Logging system information: We log several system level parameters for our internal logging purposes, i.e. NetWeaver version, Database type, etc. The function /CELONIS/FM_GET_SYS_INFO is invoked to fetch this info before the start of each extraction.
Transactions
/CELONIS/CLMAN_UI: In this transaction, we can create/delete change log tables and install triggers to store insert, update, and delete records on the source table. This is only relevant for real time extractions.
Programs
The following Programs are included in the packages /CELONIS/DATA_EXTRACTION and /CELONIS/CLTM:
/CELONIS/RP_BG_EXTRACT: This is the program that executes the extraction. It is executed via a background job.
/CELONIS/RP_BG_RM_EXTRACT: This program is called to execute the cleanup of Log tables. It is relevant only for real time extractions.
/CELONIS/RP_CL_MAN_UI: This program is used to create changelog tables and also schedule overnight clean up job to delete changelog entries past the retention period
/CELONIS/RP_CL_CLEANUP: This program is used to clean changelog entries that passed the retention period
RFC function modules
Extraction
/CELONIS/FM_NEW_EXTRACT: Read data from table, result is saved in a file.
/CELONIS/FM_CL_NEW_EXTRACT: Read data from change log table, result is saved in a file.
/CELONIS/FM_STAD_EXTRACT: Extract workload information.
Clean up
/CELONIS/FM_CL_RM_EXTRACT: Clean up change log table (after successful extraction).
Metadata
/CELONIS/FM_GET_TABLE_LIST: Retrieve DB table list using search criteria.
/CELONIS/FM_GET_CHECKED_TABLES: Verify table names.
/CELONIS/FM_CL_GET_TABLE_NAME: Retrieve change log table info.
/CELONIS/FM_CL_GET_TABLE_NAMES: Retrieve change log table info (multiple).
/CELONIS/FM_GET_TABLE_METADATA: Retrieve table column metadata.
/CELONIS/FM_GET_DPNDT_METADATA: Retrieve table column metadata, expanding appended structures.
/CELONIS/FM_GET_STAD_METADATA: Retrieve workload metadata.
Logging
/CELONIS/FM_GET_JOB_LOG: Get log messages from table /CELONIS/LOG2 from specific extraction.
/CELONIS/FM_DELETE_JOB_LOG: Clean up log table /CELONIS/LOG2 (old records).
Job management
/CELONIS/FM_GET_JOB_STATUS: Get extraction job status.
/CELONIS/FM_CANCEL_EXTRACT: Cancel running extraction job.
File management
/CELONIS/FM_GET_EXTRACT_FILE: Get extraction file content
/CELONIS/FM_GET_EXTR_FILE_LIST: Get list of files in directory Z_CELONIS_TARGET.
/CELONIS/FM_CLEANUP_FILES: Delete extraction files from specific extraction.
/CELONIS/FM_CLEANUP_OLD_FILES: Delete remaining extraction files, after the retention period is over.
Informative
/CELONIS/FM_CONFIG_TEST: Execute configuration check up.
/CELONIS/FM_GET_SYS_INFO: Retrieve Application Server info: System ID, Operating System, DB info, Celonis' RFC component version, SAP Application version, SAP Basis version, Time zone.
/CELONIS/FM_GET_RUNTIME_INFO: Retrieve memory, CPU information, and status from Application Server.