Record flattening
As Celonis Platform is designed to work with relational data, the Celonis Platform connector supports a mechanism to automatically flatten object fields nested within the source record. This can be enabled by setting connect.ems.flattener.enable to True. When enabling the built-in flattener, repeated values such as arrays and maps will be JSON encoded and persisted as nullable VARCHAR columns in the target Celonis Platform table.
Chunked JSON encoding
As part of the built-in support for flattening nested structures, we provide the connect.ems.flattener.jsonblob.chunks setting which allows us to JSON encode record payloads into a configurable amount of target table fields (chunks).
You can set the byte size of such chunks using the connect.ems.data.fallback.varchar.length. Notice that this value must be between 1 and 65000 bytes, as that is the maximum length of a VARCHAR field in Celonis Platform. This is intended as an escape hatch to allow ingesting unstructured JSON data into Celonis Platform and then implement parsing and normalization as a separate process (forexample, a custom script run at regular intervals).