Transformation task best practice
When configuring and executing your transformation tasks, we recommend the following best practice advice:
Delta transformations
For a stable and optimized data pipeline, we recommend executing delta transformations in your data model. Delta transformations process only the new or updated records from your data, rather than the full load each time. When configuring delta transformations, you need to set the primary keys for each table, allowing a MERGE to be performed.
For more information, see: Delta transformations in data jobs.
Performant scripting
The following performant scripting best practice and optimization topics are available:
SQL performance
By using optimized SQL queries, you can utilize infrastructure resources assigned to your team more effectively, and benefit from accelerated data availability of your data from source to analysis.
The following SQL performance best practice, troubleshooting, and optimization resources are available:
Transformation performance optimization
When creating transformation queries, every choice has an impact on performance. Understanding the impact gives us the possibility to apply the best practices, avoid anti-patterns and improve overall query performance.
For more information, see: Transformation performance optimization.