🎉 What’s New

We’re excited to roll out System Table-Based Extraction and Incremental Extraction for Databricks System Tables in Atlan!

These upgrades deliver:

• Faster, more accurate metadata extraction
• Optimized performance for large-scale Databricks environments
• Incremental syncs that keep metadata fresh without hitting performance bottlenecks

✨ Let’s Dig Deeper

1. System Table-Based Extraction

Atlan now supports metadata extraction directly from Databricks system tables.
This enhancement allows for deeper and more accurate visibility into your environment, especially beneficial for enterprises with vast data estates.

• Native integration for improved accuracy
• Comprehensive coverage of complex schema and lineage
• Faster extraction using Databricks' own metadata backbone

2. Incremental Extraction for System Tables

No more full crawls every time. With incremental extraction, Atlan intelligently identifies and syncs only the changed objects, such as new or modified tables.

Why It Matters:

• Faster Metadata Refresh: Your metadata stays up-to-date
• Reduced Load on Databricks: Avoid full scans, save on compute and cost
• Improved Accuracy & Coverage: Enriches metadata for advanced governance and discovery
• Scalability: Ideal for organizations managing thousands of tables or frequently changing pipelines
  
  

👏 Give It a Shot

Learn more and let us know what you think!

• System Table Extraction Method
• Incremental Extraction