🎉 What's new

The Databricks connector in Atlan now supports extracting lineage using system tables! You can leverage system tables to extract Databricks lineage, making the process significantly faster, more robust, and reducing the number of API calls required.

✨ Let's dig deeper

You can use Databricks system tables in Atlan for:

• Faster lineage extraction — instead of multiple API calls, Atlan will query system tables to extract lineage and speed up the extraction process.
• Marker-based implementation — this approach ensures a more efficient and incremental extraction of lineage.
• Reduced workflow times — faster extraction and processing lead to streamlined workflows, reducing overall workflow times.
• Incremental lineage preservation — this method of extraction supports incremental extraction and publication. This means that even if the Databricks retention period for lineage data ends, Atlan will still display lineage unless the assets are deleted within a given data flow.

👏 Give it a shot

Ready to use system tables for extracting Databricks lineage in Atlan? Get started today:

• How to set up Databricks
• How to extract lineage and usage from Databricks