Troubleshooting transformations
When configuring and running transformations, you may encounter the following issues:
Performance issues with data jobs can have a variety of root causes. Try these checks as a starting point:
Is smart ETL activated for the OCPM data job (ocpm-data-job)? To check, click on the context menu (the three vertical dots) for the OCPM data job, select Execution Settings, and check whether both toggles are activated. By default, they should be.
Are you using multiple layers of views to pre-process the data? If you decided to pre-process your data in views as described in I want to pre-process my raw data before transforming it into objects and events, keep in mind that a view is a stored SQL query that needs to be executed on every occurrence. You can think of this like a table that needs to be rebuilt with every reference of the view. If you experience performance issues in the OCPM data job (ocpm-data-job) with this setup, try using tables instead of views in the pre-processing scope, and use ANALYZE STATISTICS for the created tables. Optimization guide has more tips for optimizing the performance of tables and views.
Does your use case need multiple years of historical data? If you think your performance issue could just be a result of large data volumes, consider limiting the data scope to only a subset, as explained in Creating objects and events for a limited data scope.
To keep the SQL transformations as generic as possible, the SQL editor in the Objects and Events environment only supports a subset of functions and operators of ANSI SQL. The list of supported functions is different from that supported for case-centric data models.
You can find the list of allowed SQL functions for object-centric process mining at Supported functions for object-centric transformations. The topic includes advice on how to replace Vertica-specific functions with suitable alternatives.
Let’s look at a concrete example - in this case, the delivery is in the field VBFA.VBELN, and the delivery item number is in the field VBFA.POSNN, instead of the standard locations MSEG.VBELN_IM and MSEG.VBELP_IM that the Celonis-supplied transformations expect. You have two options:
If you don’t have the right data in MSEG.VBELN_IM and MSEG.VBELP_IM, you can create an overwrite for the DeliveryItem_ID attribute in the transformation for the OutgoingMaterialDocumentItem object type, and join the MSEG table with the VBFA table to get the missing fields. In this case, you only need to overwrite the DeliveryItem_ID attribute.
You can directly overwrite the factory for event relationships with Delivery Items.
Use a partial overwrite to overwrite individual attributes of the type. The overwrite query is joined to the query in the Celonis transformation with a left outer join.
Or use a full overwrite to completely replace the Celonis transformation for the object type or event type. A full overwrite can change IDs, attributes, joins, and filters.
Important
Fully overwriting a Celonis object or event type prevents future catalog updates from being applied automatically. For instance, if a new property is added to support a business process app, it won’t populate in an object-centric data model with a fully overwritten transformation, potentially breaking the app. If you choose a full overwrite, you must manually track and implement updates. Whenever possible, partial overwrites are preferable.
Custom attributes are defined in a single CustomAttribute script under Properties - Custom Properties, using the same ID as the main object. They publish to a separate transformation and table (x_custom_celonis_<Object Name>), which is automatically merged with the main object during data model load.
This error message means that your extracted source system data is missing one or more columns that are required by the Celonis-supplied transformations.
Sometimes the error message reports a column as missing that isn’t used in the named transformation. This can happen if you're using an imported Data Connection from another Data Pool. When you import a Data Connection to the OCPM Data Pool (as described in I have already extracted data into an existing Data Pool that is not the OCPM Data Pool), we create a view that individually selects all columns from the imported table. If one of the selected columns no longer exists, for example because it has been excluded from the extraction settings for the source table, the error message is displayed for any transformation that references the view containing the imported data. If this is the case, you can fix the error by synchronizing the imported connection as described in Sharing data between data pools.
If the columns are used in the named transformation, the best solution is to add the missing columns to your extraction configuration and do a full reload of the tables. If you cannot do a full reload, for example because of archival rules in SAP, you can still add the missing columns to the extraction configuration so that the transformations can run. Note that the new fields will not be filled retrospectively for any records that were extracted before the extraction configuration was updated. This might also have an impact on an analysis based on a perspective that uses the affected objects and events.
This error message probably means that the ID (primary key) is NULL in some of the records that are being inserted into the object type table. NULL values are not allowed for the primary key for an object type. If any of the columns that are concatenated to build the ID contains a NULL value, then the entire resulting ID is NULL.
Check the transformation referenced in the error message and investigate the source data. Most likely, at least one of the columns used contains NULL values. These values can either be eliminated during the table extraction, or removed or manipulated in a data pre-processing step using the COALESCE() function. I have NULL values in ID attributes explains how to do this.