RFC module
The entry point of the extraction process is the RFC enabled function /CELONIS/FM_NEW_EXTRACT. The extractor service passes several parameters to this function, i.e. the table, columns, filters, and other extraction metadata. The /CELONIS/FM_NEW_EXTRACT then creates a background Job, scheduling it to run the program /CELONIS/RP_BG_EXTRACT.
During an extraction, a cursor on the table that is being extracted is opened with a default buffer size of 10,000 lines (the number of lines held in memory at a time). Those lines are written to CSV files in the location specified by the FILE path Z_CELONIS_TARGET. When the CSV file exceeds a specific number of lines (default size: 50,000 lines) it is closed and then - depending on the specified compression type - compressed. Afterwards the extraction continues writing the next file.
The extractor actively polls the directory Z_CELONIS_TARGET via the RFC function /CELONIS/FM_GET_EXTR_FILE_LIST. As soon as it identifies a CSV file in the directory, the function /CELONIS/FM_GET_EXTRACT_FILE is invoked to fetch the file from the SAP file system to the extractor server. Once the transfer is complete, the file is removed from the SAP file system via the RFC call to /CELONIS/FM_CLEANUP_FILES. If the user cancels the job, the extractor calls /CELONIS/FM_CANCEL_EXTRACT and cancels the Job in SAP.
While the extraction is in progress, the extractor actively polls the background Job status via calling /CELONIS/FM_GET_JOB_STATUS. Once the Job is completed, the extractor fetches the job logs via the /CELONIS/FM_GET_JOB_LOG and publishes it to Celonis as a part of the extraction logs.
Filters are appended on the select statement of the cursor. It is important to note that arbitrary filters are possible on the tables meaning that the performance of an extraction can be influenced greatly by correct filtering on indexed columns.
Joins are not translated directly to the select query but instead each join will use its own cursor. After a chunk of the join partner has been read we use a FOR ALL ENTRIES statement for selecting from the target table. Additional filters are appended as usual.
Selecting tables/columns to extract: The user can select which tables and columns to extract in Celonis. The table list is displayed to the user via calling the functions /CELONIS/FM_GET_TABLE_LIST and /CELONIS/FM_GET_CHECKED_TABLES. To list the columns the function /CELONIS/FM_GET_TABLE_METADATA is called.
Logging system information: We log several system level parameters for our internal logging purposes, i.e. NetWeaver version, Database type, etc. The function /CELONIS/FM_GET_SYS_INFO is invoked to fetch this info before the start of each extraction.
/CELONIS/CLMAN_UI: In this transaction, we can create/delete change log tables and install triggers to store insert, update, and delete records on the source table. This is only relevant for real time extractions.
The following Programs are included in the packages /CELONIS/DATA_EXTRACTION and /CELONIS/CLTM:
/CELONIS/RP_BG_EXTRACT: This is the program that executes the extraction. It is executed via a background job.
/CELONIS/RP_BG_RM_EXTRACT: This program is called to execute the cleanup of Log tables. It is relevant only for real time extractions.
/CELONIS/RP_CL_MAN_UI: This program is used to create changelog tables and also schedule overnight clean up job to delete changelog entries past the retention period
/CELONIS/RP_CL_CLEANUP: This program is used to clean changelog entries that passed the retention period
Extraction
/CELONIS/FM_NEW_EXTRACT: Read data from table, result is saved in a file.
/CELONIS/FM_CL_NEW_EXTRACT: Read data from change log table, result is saved in a file.
/CELONIS/FM_STAD_EXTRACT: Extract workload information.
Clean up
/CELONIS/FM_CL_RM_EXTRACT: Clean up change log table (after successful extraction).
Metadata
/CELONIS/FM_GET_TABLE_LIST: Retrieve DB table list using search criteria.
/CELONIS/FM_GET_CHECKED_TABLES: Verify table names.
/CELONIS/FM_CL_GET_TABLE_NAME: Retrieve change log table info.
/CELONIS/FM_CL_GET_TABLE_NAMES: Retrieve change log table info (multiple).
/CELONIS/FM_GET_TABLE_METADATA: Retrieve table column metadata.
/CELONIS/FM_GET_DPNDT_METADATA: Retrieve table column metadata, expanding appended structures.
/CELONIS/FM_GET_STAD_METADATA: Retrieve workload metadata.
Logging
/CELONIS/FM_GET_JOB_LOG: Get log messages from table /CELONIS/LOG2 from specific extraction.
/CELONIS/FM_DELETE_JOB_LOG: Clean up log table /CELONIS/LOG2 (old records).
Job management
/CELONIS/FM_GET_JOB_STATUS: Get extraction job status.
/CELONIS/FM_CANCEL_EXTRACT: Cancel running extraction job.
File management
/CELONIS/FM_GET_EXTRACT_FILE: Get extraction file content
/CELONIS/FM_GET_EXTR_FILE_LIST: Get list of files in directory Z_CELONIS_TARGET.
/CELONIS/FM_CLEANUP_FILES: Delete extraction files from specific extraction.
/CELONIS/FM_CLEANUP_OLD_FILES: Delete remaining extraction files, after the retention period is over.
Informative
/CELONIS/FM_CONFIG_TEST: Execute configuration check up.
/CELONIS/FM_GET_SYS_INFO: Retrieve Application Server info: System ID, Operating System, DB info, Celonis' RFC component version, SAP Application version, SAP Basis version, Time zone.
/CELONIS/FM_GET_RUNTIME_INFO: Retrieve memory, CPU information, and status from Application Server.
Related topics
RFC module system requirements
Here are the requirements for a successful installation of the extractor component in the SAP system:
Minimum Version: SAP ECC 6 and Netweaver 7.0
We are actively testing against new S/4HANA versions with the aim of ensuring full compatibility and support.
If your SAP_BASIS version does not meet the minimal support package requirement, please apply the necessary Support Package or implement the corrections outlined in the following SAP Notes: 1487337, 1601030, and 1743483 (SAP Service marketplace login required).
For SAP ECC 4.6C and 5.0 we provide separate transports respectively
Here's a complete list of SAP BASIS versions with their minimal required versions of support packages necessary:
SAP_BASIS version
Minimal support package
700
SP28
701
SP13
702
SP13
710
SP16
711
SP11
720
SP08
730
SP08
731
SP05
740+
any
Development rights on the SAP system to install the modules (not required for the execution of an extraction)
Admin rights to Celonis Platform. See Managing Celonis Platform permissions.
100 GB of free disk space on the SAP system server hard drive or a network drive at the location used for the temporary data folder for Installing the RFC module.
Note
The minimum required disk space is only used for the temporary storage of the data chunks which are removed after a successful transfer. The size of the compressed chunks varies from table to table, but their size is likely below 1 GB per chunk.
The system user running the SAP system needs to have access (reading and writing) rights to the temporary data folder
Installing the RFC module
Celonis RFC module is responsible for extracting data from the SAP database based on information received from the SAP extraction client. The module gets the job metadata from the extractor client, i.e. which table, which columns, filters, etc, and then generates a background job in SAP. The job extracts the data and writes it in CSV files in a directory that is monitored by the Extractor.
Note
Installing the RFC module is a step in connecting your SAP instance with Celonis Platform
Before you begin:
Make sure you meet RFC module system requirements.
To install the RFC module, follow these steps:
In your Celonis Platform, go to Admin & Settings > Download Portal.
Download the latest version of the RFC module that fits your system.
In your SAP instance, check the folder where transports are placed with the transaction AL11 and look under DIR_TRANS.
Unzip the ZIP file with the RFC module.
From your unzipped package, copy the cofile (K...) into DIR_TRANS/cofiles and the data file (R...) into DIR_TRANS/data.
In SAP, run the STMS transaction:
Click on import overview (F5).
Double-click on the import queue of the target system
Click Extras > Other Requests > Add.
Enter the name of the transport which is AI1K... (where "..." should be replaced by the number that is contained in the file names of the two transport files).
Figure 1.
Example: adding transport request to import queue.
Click Import.
Note
If you're using the RFC module version 3.9 or later, you can skip this step. With newer versions of the RFC module, a staging table inside the SAP database is used for storing temporary files. Using a staging table for data extraction must be manually enabled in the advanced settings of your data connection. For details, Extracting data using staging .
Create a dedicated directory on the file system of your SAP (preferably a network drive) and make sure that the user running the SAP system has read and write access to it.
Warning
In case you are using Logon Groups, or there are multiple application servers where the extraction jobs can run, make sure that the directory is on a network drive, and is accessible from all the servers.
In your SAP system, run the FILE transaction.
Find the Logical Path Z_CELONIS_TARGET in the list
Edit the entry by clicking on the button to the left of the new entry and then double-click on the folder "assignment logical and physical paths"
Set a path for UNIX and/or Windows. The path should be the directory on the system or a network drive that you created in Step 1.
Make sure to include the <FILENAME> placeholder at the end of the directory.
Z_CELONIS_TARGE for multiple clients
When setting up the physical path for Z_CELONIS_TARGET, you can use a dynamic <CLIENT> parameter for client number. For example:
tmp/<CLIENT>/CELONIS/<FILENAME>
On runtime, the <CLIENT> parameter will be replaced with the client number for which the extraction is running. For example:
tmp/100/CELONIS tmp/800/CELONIS
You now have to configure user permissions for SAP connection. See Create users for SAP connection.
Enable near real-time data extraction and analysis of SAP systems. See Set up SAP real-time extension.
Related topics:
Updating RFC module and SAP extractor
The RFC module and SAP extractor are regularly updated, requiring you to update them as part of your connection between your SAP source system and the Celonis Platform.
You can update the RFC module you have installed in your SAP system by downloading the latest version from the Celonis Platform download portal. Once downloaded, you must then import the module into your SAP system.
When updating your RFC module, we also recommend that you update your extractor. While the RFC module is backward compatible, using the latest version gives you access to all available features and fixes.
Checking your current RFC module version
You can check which RFC module version you are currently using from the data connections screen by clicking Options - Extractor Information:
 |
Updating RFC module
To update your RFC module, you need admin access to your Celonis Platform team. With admin access, follow these steps:
From the Celonis Platform, click Admin & Settings.
Click Download Portal.
Locate and download the RFC Module for SAP ECC and S/4HANA.
From the SAP system, import the transport.
For more information, see 2. Import the transport.
Restart the extractor by using the instructions outlined in How do I set up an on-premise Extractor?
What's next
Here are some things you might need to do after you've updated the RFC module:
If you're upgrading the RFC module from any earlier version than 3.6.0, you must also upgrade the Celonis SAP user role. This ensures that your RFC module has the correct permissions to perform the necessary tasks.
To see how to edit roles for existing Celonis SAP users, see Create users for SAP connection.
If you are upgrading from an RFC Version earlier than 2.0, and the real-time mode is on, you must delete and recreate the Change Log tables in the SAP system.
Before updating the SAP extractor, make sure that all the current requirements of the extractor server are fulfilled, see How do I set up an on-premise Extractor?
Updating with installer (Windows only)
Updating the extractor using the installer is only possible if the existing instance of the extractor was also installed with the installer. If that's not the case for you, a clean installation will be executed instead of an update.
To update the Extractor using the installer:
Download the latest installer and execute it.
This will replace the sap-connector.jar file with the newest one, and restart the Celonis Windows Service.
Manual update
To update the SAP on-premise extractor, you need to perform the following steps:
Download the latest Extractor package from the Download Portal.
Stop the current extractor if it is still running.
If necessary, setup the JCo Library. Download it from the SAP Marketplace, and copy the files below to the folder "Extractor path\jco" on the extractor server.
sapjco3.jar
sapjco3.[.dll | .so | .sl ] (file extension depends for which operating system the library was downloaded)
Replace the <extractor name>.jar file with the new one.
Restart the extractor by using the instructions outlined in How do I set up an on-premise Extractor?
To revert your RFC module version to a lower one:
In Celonis Platform, go to Admin Settings > Download Portal.
Download the Celonis_RFC_Data_Extraction_x.y-UPDATE.zip package.
Install the RFC module using the downloaded package. See Installing the RFC module.
Your downloaded version of the RFC module will overwrite the previously installed version. If you had any change log tables, they will have to readded manually. See Adding change log tables.
Troubleshooting RFC module installation
We've gathered some resources and troubleshooting tips that will help you resolve some problems you might encounter when installing the RFC module.
Symptom
When importing the RFC Module into your SAP system you may get the error message: "Does not match component version" or "Checking components of the requests" as s hown below.
 |
Cause
With Version SAP_BASIS 702 a new feature was added to the TMS system to check the component version of the transport which was released and the component version of the system where the transport was going to be imported. If there is a difference, this difference is highlighted in the import queue (transaction stms_import) and if you try to import the transport, the import will stop with a warning message about the difference in the SAPCOMPONENT versions.
Solution
Ideally, the source system and target system should match in relation to component versions, but this is not possible for the RFC Module, because Celonis has only 1 development system, while our customers have a multitude.
You can bypass the warning with the import option "Ignore invalid component version". You will see this option in the options tab upon import.
 |