Record flattening
As EMS is designed to work with relational data, the EMS connector supports a mechanism to automatically flatten object fields nested within the source record. This can be enabled by setting connect.ems.flattener.enable to True. When enabling the built-in flattener, repeated values such as arrays and maps will be JSON encoded and persisted as nullable VARCHAR columns in the target EMS table.
Chunked JSON encoding
As part of the built-in support for flattening nested structures, we provide the connect.ems.flattener.jsonblob.chunks setting which allows us to JSON encode record payloads into a configurable amount of target table fields (chunks).
You can set the byte size of such chunks using the connect.ems.data.fallback.varchar.length. Notice that this value must be between 1 and 65000 bytes, as that is the maximum length of a VARCHAR field in EMS. This is intended as an escape hatch to allow ingesting unstructured JSON data into EMS and then implement parsing and normalization as a separate process (forexample, a custom script run at regular intervals).