Supporting content for SAP ECC and SAP S/4HANA data connections
The following sections can be used when configuring your connection between SAP ECC and SAP S/4HANA and the Celonis Platform:
Connection Test
When you test a connection, the following steps are performed:
Check whether all the function modules exist
Write an empty file on the SAP server in the defined folder
Compress the newly created file (if compression is enabled)
Send the file to the extractor
Therefore the following aspects are tested:
Existence and completeness of the RFC module
Writing rights on the SAP server in the respective folder
Reading rights on the SAP server in the respective folder
Correctly configured compression method
Open firewall ports to send RFC call to the SAP system
Open firewall ports to send files from the extractor to the cloud endpoint
Connection Test Troubleshooting
Below you can find the errors that may cause the connection test to fail and the respective solution(s), grouped by the connection checklist task.
When a checklist task fails in the SAP connection test, you can click on the button Help, to navigate to the relevant solutions in this page.
 |
SAP JCo not found on classpath, ensure 'sapjco3.jar' exists
Solution
The JCo library Java part (sapjco3.jar) needs to be placed within the 'jco' folder of the extractor. You can download the respective package from the SAP Marketplace.
Note: This step should be performed by someone who has an SAP Service User (S User), which authorises to download software from SAP portals. Usually the SAP BASIS has such access.
A restart of the Extractor is required after the installation.
SAP JCo native library not found
Solution
The JCo native library file sapjco3 (.dll / .so / .sl / .dylib) needs to be placed within the 'jco' folder of the extractor. You can download the respective package from the SAP Marketplace.
Note: This step should be performed by someone who has an SAP Service User (S User), which authorises to download software from SAP portals. Usually the SAP BASIS has such access.
A restart of the Extractor is required after the installation.
Unsupported operating system for SAP JCo native library
Solution
The JCo native library file sapjco3 (.dll / .so / .sl) is not supported by your operating system. Make sure to place the correct library file for your respective operating system.
A restart of the Extractor is required after the installation.
Solution
The Visual 2013 C++ Redistributable Package needs to be installed on the server.
A restart of the Extractor is required after the installation.
The connection between the Extractor Server and SAP could not be established
Solution
Check if the Host and the System Number are specified correctly in the Data Connection.
If you are using Message Server as middleware, check if the Message Server Host and the Message Server Port are specified correctly.
Check if the following network connectivity requirements are satisfied:
Source System | Target System | Port | Protocol | Description |
|---|---|---|---|---|
On premise extractor server | SAP ECC system | 33XX (where xx is the system number) | TCP | RFC connection from on premise extractor server to the SAP system. The system number can be retrieved from the SAP basis team. |
Error during connection validation
Solution
Please check the specific error message that is provided in addition.
If this does not help, open a support ticket.
Missing authorisations for the Celonis RFC User
Solution
The Celonis RFC user is missing required authorizations. Make sure you have downloaded the dedicates user role from the Download Portal.
Afterwards follow these steps to apply the correct role to the technical user of the RFC connection:
Run the transaction PFCG
Go to Role → Upload
Select the provided role file (CELONIS_EXTRACTION.SAP)
Apply the uploaded role CELONIS_EXTRACTION to the RFC user
If you wish to set up the user yourself, make sure that the user has the following permissions:
Permission to write on the server hard disk under the path specified in step B
Permission to execute all RFC modules contained in the transport imported in step A (package name /CELONIS/DATA_EXTRACTION )
Read access to all tables that should be extracted and DD02L and DD03L for retrieving the table and column names
Necessary Celonis Function not implemented in SAP
Solution
You have not installed the Celonis RFC Module or you have an outdated version in place.
For all the installation files, go to the Download Portal.
Please update your RFC Module to use the 'Zip' compression method
Solution
The minimum requirement isRFC Module version 1.6
For all the installation files, go to the Download Portal.
RFC Module call failed
Solution
The called RFC function resulted in an internal error.
You can check the specific error message. If that doesn't help to fix the error, open a support ticket.
Your RFC module version does not support usage of native SAP compression. Please update your RFC moduleto use this feature
Solution
The minimum requirement is RFC version 1.6. As Extractor version 2019-11-12is required.
For all the installation files, go to the Download Portal.
In case an upgrade is not possible, you can switch to gzip compression.
Checking writing of parquet files
Solution
The Extractor Service is lacking permissions to write to the 'temp' folder in the extractor directory.
If the permissions are fine, open a support ticket.
Test file writing failed
Solution 1: (Z_CELONIS_TARGET is set to its default value):
Create a dedicated directory on your SAP server (preferably a network drive) and make sure that the user running the SAP system has reading and writing access to it
Warning
In case you are using Logon Groups, or there are multiple application servers where the extraction jobs can run, make sure that the directory is on a network drive, and is accessible from all the servers.
Run the transaction FILE in your SAP system
Find the Logical Path Z_CELONIS_TARGET in the list
Edit the entry by clicking on the button to the left of the new entry and then double-click on the folder "assignment logical and physical paths"
Set a path for UNIX and/or Windows (shown in the screenshot on the left); the path should be the directory on the system or a network drive that you created in step 1
Solution 2:
The logical path Z_CELONIS_TARGET is not accessible from the application directory.
If you are using load balancing, please make sure to use a network drive instead of a local directory.
Missing authorizations for the Celonis RFC User
Solution
The Celonis RFC user is missing required authorizations.
Afterwards follow these steps to apply the correct role to the technical user of the RFC connection:
Run the transaction PFCG
Go to Role → Upload
Select the provided role file (CELONIS_EXTRACTION.SAP)
Apply the uploaded role CELONIS_EXTRACTION to the RFC user
If you wish to set up the user yourself, please make sure that the user has the permissions below.
Permission to write on the server hard disk under the path specified in step B
Permission to execute all RFC modules contained in the transport imported in step A (package name /CELONIS/DATA_EXTRACTION )
Read access to all tables that should be extracted and DD02L and DD03L for retrieving the table and column names
Test file permissions failure
Solution
Please make sure that the operating system as well as the SAP system have reading and writing permissions for the logical path Z_CELONIS_TARGET.
Test file compression failed
Solution
Make sure the command (for your compression method) is configured in SM69. Check if the command is valid and that an executable exists.
Compressed test file not found
Solution
Check if the command (for your compression method) configured SAP transaction SM69 is configured correctly.
Test file deletion failed
Solution
Make sure to download the Celonis User Role from the Download Portal and add it to the RFC user.
Z_CELONIS_TARGET is set to its default value. Please check there is enough space in this location, and if you have multiple servers configure it to a network share
Solution
Create a dedicated directory on your SAP server (preferably a network drive) and make sure that the user running the SAP system has reading and writing access to it
Warning
In case you are using Logon Groups, or there are multiple application servers where the extraction jobs can run, make sure that the directory is on a network drive, and is accessible from all the servers.
Run the transaction FILE in your SAP system
Find the Logical Path Z_CELONIS_TARGET in the list
Edit the entry by clicking on the button to the left of the new entry and then double-click on the folder "assignment logical and physical paths"
Set a path for UNIX and/or Windows (shown in the screenshot on the left); the path should be the directory on the system or a network drive that you created in step 1
Listing files failed
Solution
The application server cannot access Z_CELONIS_TARGET.
Make sure to download the Celonis User Role from the Download Portal and add it to the RFC user.
On average, you should expect that the following extraction throughputs:
Large tables - such as BSEG or 100k records/minute.
Mid size - such as EKKO, VBAK, or 250k records/minute.
However, these numbers are just an approximation and the actual duration can vary significantly depending on the source system. To understand the variance, it is important to understand the overall flow and services involved:
The extraction is based on an SQL query, which is executed at the database level.
The output is stored in the SAP memory.
From memory, the data is written to .csv files.
These .csv files are then fetched by the On-Premise-Client to be uploaded to the cloud.
A problem in any of these steps will impact the performance of the overall runtime. The following are the most common factors that can impact performance:
Database and SAP server capabilities
The more resources the database and the application server have, the faster the extraction query will run.
Source system load
Running the extractions during the busy hours will lead to slower performance since there are more requests competing for the same resource pool.
Complexity of the applied filter and table size
This is a very important and often neglected factor. If you apply a filter to non-primary keys, the runtime will be way slower than with a PK-based filter which is indexed in the database. The table size also plays a big factor, especially with non-PK filters. Extracting 1 million records out of 100 billion won’t have the same duration as extracting 1 million out of 100 million.
Size of the table
Depending on the business, the size of the same table can vary significantly. This will lead to different runtimes for the same table, even if the number of extracted records is the same.
Network bandwidth between the OPC and SAP Servers
OPC fetches the files from the SAP server, and if the network has a low bandwidth, this will impact the data transfer and consequently the overall extraction speed.
When upgrading from SAP ECC to S/4 HANA, the Celonis data pipeline may be affected, particularly in the Extraction and Transformation layers. Below, we explore various scenarios and their expected impacts.
Celonis RFC Extractor
The RFC module is an ABAP application that is compatible with both SAP ECC and S/4 HANA. It uses no version specific functionality and runs seamlessly in both systems without any migration. The only exception is when the Real-Time Extension is activated, in which case a migration is needed.
Real Time Extension is not active
Minimal migration effort is needed in this scenario. The RFC Module is used only for batch-based extractions and there are no Celonis change logs or triggers installed in SAP.
In this scenario follow these steps:
Confirm the successful transfer of the RFC Module in S/4 HANA. Search for the package /CELONIS/* (the full name varies depending on the extractor version).
Set up the target path Z_CELONIS_TARGET as explained in Installing RFC Module.
Check that the Celonis user exists in S/4 and has the appropriate permissions.
Real Time Extension is active
In this scenario, change log tables and triggers installed in SAP ECC need to be recreated in S/4 HANA to accommodate table schema changes. Recreating them will resolve this issue. In general, we advise that you perform the “Real Time Extension” setup from the scratch by following the steps in the Set up and Configuration of Real-Time Extractor section of the "Real Time Extension" topic. Follow this step carefully to avoid issues during the upgrade.
Warning
In S/4 many tables have been replaced with compatibility views, i.e. MSEG. Since they no longer physically exist in the database, you won’t be able to install triggers on them. To extract these tables in real time you need to switch to the replacing table, i.e. MATDOC instead of MSEG.
Extraction and Transformation configuration in the Celonis Platform
S/4 HANA comes with compatibility views so that every deprecated SAP ECC table has a corresponding view in S/4. As a result, old integrations won’t break and can also be run against S/4. This also applies to the Celonis Platform, meaning that you can keep running the same Extraction=>Transformation against S/4 and expect the same dataset.
However, if the real-time extension is active and you have to replace tables that don’t exist anymore, then you have to modify the Extractions as well. In the MSEG example, you need to replace it with MATDOC. You can also rename it to MSEG during the extraction in order to keep the transformations intact downstream.
When starting an extraction in the Celonis Platform, every table being extracted maps to one background job in the SAP system.
The number of parallel extractions (and therefore background jobs) can be modified in the connection details of the data connection.
To see the jobs created by the Celonis Platform in SAP, you can go to run transaction SM37. To find the correct jobs, enter the prefix of the created job names (CEL_EX*) into the job name field and the name of the Celonis RFC user in the user field.
If one job on this day already ran, you may see a listing like on the left.
It shows the successful extraction of the table T001 from the Celonis Platform.
In this case the BSEG was running from one Data Job, the DD0* Tables from another job.
When cancelling one of the data jobs, the overview may look like the following.
The job extracting the DD0* Tables was cancelled from the Celonis Platform, the job extracting the BSEG is still running.
Cancelling a Job from the SAP System
If we want to cancel a job from the sap system, we can simply do so from the transaction SM37.
In the Celonis Platform, we can look at the job log and see something like the picture on the left. So the cancelling in SAP is picked up by the Celonis Platform and the Data Job is set to failed.
To remove the Celonis Extractor from SAP system perform the following steps:
(Optional) Stop all replications and/or the extraction data jobs from the Celonis cloud.
(Optional) If the Real-Time Extension is active:
Delete the Triggers via the transaction code /CELONIS/CLMAN_UI in DEV, QA and PROD;
Delete the Change Log tables in DEV and then move the transport forward to the QA and PRD systems;
Remove the CEL_CL_CLEAN_UP_SCHEDULED clean up job, if it was set up, in all systems.
Download the latest “Uninstall transport” for the RFC Module from the Downloads portal and import it into SAP.
This transport removes all objects shipped in the RFC Module.
After running it, verify that nothing exists in the /CELONIS/namespace.
Delete the Celonis user and role from SAP.
We are generating and storing detailed extraction logs on the Extractor server locally. By default they are stored in the root folder where the file connector-sap.jar is located. A new log file is generated on a daily basis, and with the time the number of logs can grow up significantly. To prevent this, the user can set up a retention period in the file application-local.yml, and the older logs will be cleaned up automatically.
In the file application-local.yml, add the following line in the "logging" section, as shown below. Make sure that the spacing is correct.
Restart the Extractor service