🎉 What’s new

We’re excited to unveil significant enhancements to the Databricks connector, elevating the crawler and miner capabilities to drive greater flexibility, governance, and trust across your Databricks landscape.

✨ Let’s dig deeper

• Primary and Foreign Key (PK & FK) Support: We now capture and display PK/FK information from Databricks, helping users identify the unique keys of assets.
• Materialized View Support: We now ingest and catalog Materialized Views created via native Databricks SQL. Users can visualize MV lineage and gain clear insights into their dependencies and upstream sources.
• 
  Enriched DBX Lineage Process: Users can now view enriched Databricks lineage with detailed entity types (e.g., Pipelines, Notebooks) and unique entity IDs, providing clarity and traceability in their data flows.
• Enabling Cloned Catalog Support for Databricks Miners: We now allows configuring alternative catalog for Databricks Miners when access to the default System tables is restricted. This provides controlled access to lineage metadata, ensuring that users only see information relevant to their business entity.
• Improved Preflight Checks: Enhanced preflight validations provide a smoother Databricks onboarding experience in Atlan.
  

👏 Give it a shot

Learn more and let us know what you think!

• How to set up Databricks 
• What does Atlan crawl from Databricks?
• Create cloned views of system tables
• Troubleshooting Databricks connectivity 

🎉 What’s new

The Atlan Secure Agent is a lightweight application that is deployed within an enterprise and facilitates metadata extraction. The Secure Agent acts as a gateway and connects to source systems and extracts metadata. This architecture avoids inbound connectivity to these systems as well as ensures secrets are managed internally. 

✨ Let’s dig deeper

The Secure Agent is designed for secure, scalable, and efficient metadata extraction.

Currently supported connectors

• Oracle, MS SQL Server, PostgreSQL, Salesforce
• More connectors getting supported soon!

Security-first architecture

• Runs entirely within the organization's infrastructure, preventing secrets from leaving its boundary.
• Uses outbound, encrypted communication to interact with Atlan SaaS.
• Supports logging and monitoring and integrates with external monitoring systems for auditing and compliance.

Scalable metadata extraction

• A single deployment of the Agent can connect to multiple source systems.
• Supports multiple concurrent metadata extraction jobs.

Flexible deployment

• Deploys on cloud-based Kubernetes environments (such as Amazon EKS, Azure AKS, and Google GKE) as well as on-premises clusters.
• Scales dynamically based on workload demands.

Automated operations

• Continuously monitors system health and sends heartbeats to Atlan.
• Captures and uploads execution logs for troubleshooting and auditing.

👏 Give it a shot

Switch on the Labs toggle to enable Agent functionality on the tenant.

Go through the Agent installation and configuration docs and setup a workflow in Agent execution mode

Feel free to reach out to your Atlan representative to know more!

🎉 What’s new

We are excited to announce the addition of QVD file support in Qlik! With this new feature, users can now:

• Crawl QVD datasets: Directly crawl and explore datasets created by QVD files within Atlan.
• Discover Data: Perform data discovery around these datasets to gain insights and enhance data-driven workflows.
• Lineage Support: We’ve also added lineage support for datasets generated by QVD files, ensuring a complete understanding of how data flows and evolves across your systems.
  

✨ Let’s dig deeper

• QVD File Datasets as Regular Assets:Datasets obtained from QVD files will now be treated as regular dataset assets, making them easier to manage and track within the system.

• New Dataset Attribute for Implicit Status: A new attribute will be added to indicate whether a dataset is implicit or not. This will help clarify the nature of datasets and their role in the overall usage.
• Chart Asset Type Filtering:We will now filter out all non-relevant entities from the chart view, displaying only the chart asset type for a cleaner, more focused visualization.

👏 Give it a shot

Learn more and let us know what you think!

• What does Atlan crawl from Qlik Sense Cloud
• Troubleshooting Qlik Sense Cloud connectivity

🎉 What’s new

We are excited to announce the addition of SCRAM authentication support for the Apache Kafka connector. This feature allows for a more secure form of password-based authentication.

• SCRAM Authentication: Kafka now supports SCRAM-SHA-256 and SCRAM-SHA-512 for authenticating clients and brokers.
• Improved Security: SCRAM provides stronger password hashing and eliminates the need for plaintext passwords, ensuring better protection against brute-force attacks.
  

✨ Let’s dig deeper

This release makes it easier to ensure a higher level of security in your Kafka environment, reducing the risk of unauthorized access and improving overall system integrity.

• Enhanced authentication security for Kafka clusters.
• Better integration with existing enterprise security practices.
• Allows fine-grained access control through password management.

👏 Give it a shot

Use a more secure way to integrate Apache Kafka with Atlan!

Learn more:

• How to set up Apache Kafka

• How to crawl Apache Kafka

  
  

🎉 What's new

With this update, we’ve introduced several improvements to the workflow experience:

• Users can now add a custom name to workflows.
• We now display the user who has stopped a workflow for better traceability.
• The Run Workflow button will be disabled if a workflow is already running, preventing overlapping executions.

These changes enhance your workflow management and improve overall usability.

✨ Let's dig deeper

• Improved Workflow Management: Allowing users to add custom names to workflows provides better organization and easier identification of workflows, especially in complex processes.
• Enhanced Traceability: Displaying the user who stopped a workflow increases accountability and helps track the source of any interruptions, making it easier to troubleshoot and manage workflows.
• Prevention of Overlapping Executions: Disabling the Run Workflow button when a workflow is already running prevents accidental overlapping executions, ensuring smoother operations and preventing potential errors.

Together, these enhancements streamline workflow management, improve collaboration, and reduce the risk of operational issues.

👏 Give it a shot

Check out these enhancements and let us know what you think!

🎉 What's new

You can now use the display name of your custom metadata attributes to update your dbt assets in Atlan!

Atlan now supports two ways to enrich custom metadata for your dbt assets programmatically:

• businessAttributes — supports the existing method of using hashed-string names.
• businessAttributeNames — accepts display names for custom metadata in a human-readable format.

Enjoy additional flexibility to collaborate with key stakeholders as you enrich your dbt assets in Atlan.

Learn more: Change custom metadata on assets

🎉 What's new

Atlan now supports nested columns 15 levels deep for AWS Glue! We've expanded our support from level 1 to level 15 for AWS Glue nested columns, empowering you to go deeper into your data.

✨ Let's dig deeper

Here's what you need to know:

• Atlan retrieves raw STRUCT and ARRAY type objects for AWS Glue nested columns up to 15 levels.
• You can search for and discover nested columns like any other column asset in Atlan.
• You can view nested columns in the column preview and overview sidebar for your table assets.
• You can enrich your nested columns with tags, certificate, description, and more.
• Column-level lineage is supported, so you can view lineage for nested columns on the lineage graph.
• Tag propagation is currently only supported from parent to nested columns.
• Atlan currently does not parse MAP type objects for columns and nested columns.

👏 Give it a shot

Discover and document your nested columns from AWS Glue!

Learn more: What does Atlan crawl from AWS Glue?

🎉 What's new

If you update or rename your Airflow tasks at source, Atlan will delete the old cataloged tasks and associated processes and then reflect the latest state on both the lineage and pipeline graphs. With your data estate always evolving, this release ensures that Atlan is always in sync with your latest updates. The processes created for the task will be updated to reflect these changes as well.

🎉 What's new

Here are some hot-off-the-press improvements and fixes for our dbt users in Atlan:

• dbt test mapping in asset preview and sidebar — we've fixed dbt test mapping in the asset preview card and sidebar. Previously, dbt test results did not display the right environment relationship between the materialized table and dbt test, displaying incorrect information. We've fixed the test mapping to show the correct environment.
• Descriptions for dbt materialized assets — we've updated the way in which Atlan pulls descriptions for materialized tables. If a materialized table is linked to multiple dbt assets, Atlan will apply the description from the linked dbt model to the materialized asset. In case this is unavailable, Atlan will then apply the description from the linked dbt source to the asset.
• Greater visibility for job and model statuses — to provide you with full clarity, Atlan now breaks down job and model status for your dbt materialized tables. You can view the model that materialized a specific table and model status or the job that executed the model as well as job environment and status.
  
  

🎉 What's new

Atlan now crawls Monte Carlo alerts and incidents for the last 30 days by default! This ensures that the red indicators and failing monitors are relevant and actionable, helping you focus on recent and meaningful issues. You'll also have the option to set the date range to the last 14 or 45 days instead of the default configuration.

✨ Let's dig deeper

Previously, Atlan fetched all historical alerts and incidents from Monte Carlo, which often resulted in irrelevant red indicators and failing monitors. For example, an outdated failing monitor from one year ago could show up in Atlan today, causing unnecessary confusion.

Hence, we've optimized the Monte Carlo integration to deliver the following:

• Faster syncs — reduced runtime for the Monte Carlo package, enabling you to schedule your workflows more frequently.
• Fewer API calls — a significant reduction in API requests to avoid any rate-limiting issues with Monte Carlo.

👏 Give it a shot

For any new and existing Monte Carlo integration in Atlan, you'll only receive alerts and incidents for the last 30 days after the next workflow run. You can also modify the configuration to change the data range.

Learn more: How to crawl Monte Carlo