🎉 What’s new

Lineage is how people and AI agents answer “where did this number come from?” and “what breaks if I change this?” 

More accurate edges mean impact analysis you can act on. Reliable processing at scale means a complete graph. And lineage that respects your access policies means you can hand it to AI with confidence.

Over the past three months we’ve worked on all three.

Lineage in Atlan is now more accurate, holds up at enterprise scale, and is safe for AI agents to explore, whatever connectors you use. Every change below sits in the lineage layer that people, Atlan AI and your own agents share.

Let’s dig deeper

Built for enterprise scale

• Big backfills finish, and stay finished. Lineage built from query history now runs one day at a time, and each day saves its progress. A failure costs one day, not the whole run. One recent onboarding processed dozens of days of history and 1.5+ billion relationships in ~ 12 hours with no restarts. The same job used to time out.
• Sharper SQL parsing on every connector. A new SQL parsing engine now runs for all connectors. Tables with the same name in different schemas stay distinct, and tricky SQL such as SELECT * expansion and correlated subqueries resolves correctly.
• Dense graphs show the full picture. Open a grouped edge and the lineage drawer lists every process behind it, not just the ones loaded on the canvas. When View Impact hits a very large graph, the CSV download now uses the next-gen impact report.

Built with privacy in mind

• Query redaction keeps sensitive values out of Atlan. Turn on Redact SQL Text and literal values in captured SQL, such as customer IDs, emails and amounts, become placeholders before the SQL is stored. Table and column references stay, so your lineage stays intact. SQL that can't be parsed is kept as-is. Available today for Snowflake, BigQuery, Oracle, SQL Server, Cloud SQL for PostgreSQL, AlloyDB for PostgreSQL, Looker and Tableau, with more connectors coming.
• AI agents only see the lineage you can see. When Atlan AI, or any agent connected through Atlan MCP, traces lineage, it follows your access policies. Assets you can’t read stay hidden: their names, owners, descriptions and SQL never reach the model. This applies to all lineage in Atlan, from every connector.

Connector highlights
We know you care about the few connectors at the heart of your stack. While we are making improvements every single day across hundreds of connectors, here are some sample lineage enhancements for popular systems!

• Apache Airflow: lineage for each query. When one task runs several SQL statements, such as BigQuery scripts, stored procedure CALLs or hook SQL, each statement now appears as its own child task with its own lineage. No more cross-linked edges: each target shows only its real source. The lineage drawer also shows the DAG’s schedule instead of “Manual run”. Requires Airflow 2.11+, apache-airflow-providers-openlineage 2.16.0+ and apache-airflow-providers-google 22.3.0+.
• Google BigQuery: lineage across connections. Lineage now continues across connection boundaries, at both table and column level, instead of stopping at the edge of each one. The new Standard Lineage workflow joins lineage across the connections you pick, one connector at a time, at both table and column level. A query in one connection now links to tables crawled by another. BigQuery is first. Table Valued Functions are also now a part of lineage graph for BigQuery.
• Microsoft Power BI: up to 96% faster extraction. We removed an expensive API call, which cuts extraction time and your Premium capacity use with no loss of lineage coverage. Lineage now stays stable from run to run, reaches Databricks sources for both datasets using Databricks.Query and dataflows, links paginated reports to the semantic model they read, and column-to-page lineage now covers every field a visual uses, including fields used only in filters.
• Microsoft Fabric: Dataflow Gen2 on the lineage graph, with column-level lineage passing through. Semantic model measures are now searchable assets with their DAX (beta), and Spark notebook lineage is in private preview.
• Amazon Redshift: stored procedures, with every statement credited to the procedure that ran it.
• Amazon QuickSight: field-to-field lineage, from source dataset fields to the exact field each analysis visual uses.
• Snowflake: lineage now draws on every query type in query history.
• Dremio: column-level lineage, including through virtual datasets.
• Confluent Kafka: lineage through Confluent Cloud Connect connectors.
• Qlik Sense and MicroStrategy: both rebuilt with SQL lineage. Qlik Sense also gains Enterprise on Windows support.
• Tableau and Sigma: lineage stays stable from run to run, so far fewer phantom “changed” processes.

Give it a shot
Most of these improvements are already live in your lineage graph, with nothing to turn on, but check the docs links or reach out to your Atlan contact with any questions, and let us know what you’d like to see next!