Lineage
Recent items mentioning Lineage across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Recent large-scale implementations rely on end-to-end data lineage for audit-grade accountability, spanning California's unified behavioral health tracking across 58 counties 1 and Dow's third-party-assured Carbon Footprint Ledger 4. Databricks guidance emphasizes that platform-native, automated lineage in Unity Catalog prevents redundant metadata layers 25 while providing the governance trail required to operationalize high-stakes financial AI models like credit scoring and fraud detection 3.
Generated daily from the 7 most recent items mentioning Lineage. Click any [N] to jump to the source.
How California built its Behavioral Health Public County Profile on Databricks
California launched the Behavioral Health Public County Profile, a public-facing dashboard powered by Databricks that unifies fragmented behavioral health data across all 58 counties. Built to support the Behavioral Health Services Act, the platform establishes standardized metrics, robust governance, and end-to-end data lineage across disparate county systems to transparently track local funding, service delivery, and homelessness initiatives.
Choosing Data Governance Tools for Enterprise Data Governance
Governance tools enforce access control, quality checks, and lineage tracking at the platform layer, but their architecture ranges from standalone catalogs to platform-native suites, so matching the category to your underlying data stack is what prevents duplicate metadata layers and policy gaps. Evaluation criteria like policy enforcement granularity and AI/agent governance differ widely across tools, a gap that matters more as autonomous agents begin querying production data directly.
AI Applications in Finance: A Practical Use Case Guide
Scaling AI across core financial operations like credit scoring and fraud detection requires disciplined governance, including explainable models, documented data lineage, and human-in-the-loop checkpoints. A phased framework centered on two prioritized pilots, a 90- to 120-day evaluation window, and rigorous ROI measurement provides teams with a low-risk path to enterprise-wide deployment.
How Dow Built a Carbon Footprint Ledger on Databricks to Accelerate Sustainability at Scale
Dow built an enterprise Carbon Footprint Ledger on the Databricks Data Intelligence Platform, collapsing cradle-to-gate Product Carbon Footprint calculation times from weeks to a fraction of that across its entire portfolio. Powered by Apache Spark, Delta Lake, Unity Catalog, and MLflow, the solution delivers full data lineage and audit-grade governance designed for third-party assurance against ISO 14067 and GHG Protocol standards.
TutorialsDatabricks Unity Catalog: The Safe Way to Govern AI
Databricks Unity Catalog provides a single governance layer for data and AI assets, enabling discovery, classification, protection, and certification of data. It demonstrates how to use Unity Catalog for context-aware search, automated lineage tracking, tagging sensitive data with govern and non-govern tags, and applying column masking for data protection.
Data Governance Architecture: A Complete Blueprint for Modern Organizations
This blueprint details a complete data governance architecture, outlining the policies, roles, and technologies needed to manage data assets. It emphasizes a modern strategy combining automated lineage, RBAC, and federated models to ensure data quality and regulatory compliance at scale.
TutorialsThe Future of Finance Operations Starts Here
The video demonstrates how Databricks' financial lakehouse solution addresses common finance data challenges like fragmentation and slow analysis. It showcases features like Unity Catalog for data governance, Lake Flow for pipeline management, and Genie Spaces for natural language querying of financial data.
Ship smarter agents in production with dbt Agent Skills
dbt Labs has released an open repository of dbt Agent Skills, porting core analytics engineering best practices like data lineage, contracts, and iterative testing into Anthropic's open agent protocol. By packaging analytics development lifecycle workflows into reusable skills, teams can prevent broken runs and safely automate dbt model development with generalist coding agents.
AI Applications: Tools, Use Cases, and Platforms
AI applications span four capability tiers, each with distinct data requirements and evaluation frameworks, and enterprise deployments often stall due to inadequate data infrastructure. Production-grade model development, from prompt engineering to pretraining, is increasingly accessible with open-source LLMs, but requires pre-built governance and monitoring infrastructure for successful deployment at scale.
How AI improves data lineage at scale
AI copilots solve data lineage scaling bottlenecks by accelerating transformation code development, generating contextual node documentation, and automating test suite creation. This approach replaces guesswork in impact analysis with reliable, well-documented DAGs that remain accurate and maintainable as analytics projects scale.
TutorialsHow to use Recursive CTEs in Databricks
The video demonstrates how to use recursive CTEs in Databricks to traverse hierarchical data structures of unknown depth, such as data lineage or organizational charts. It shows how to write a recursive CTE in SQL, highlighting the `RECURSIVE` keyword and the union of an anchor member and a recursive member.
NewsFrom Raw Data to Real-Time Retention: Powering Customer Health Scores on Databricks
Coales is a data development tool built on Databricks that uses templated pipeline components and AI-assisted discovery to enable rapid data transformation without extensive custom coding. Coales deployed their own tool to automate customer health metrics scoring, replacing manual analysis with a real-time self-service pipeline for their sales and professional services teams.
TutorialsDatabricks Unity Catalog Tutorial - Data Governance made easy
Unity Catalog provides data governance capabilities including data discovery, access control, auditing, and data lineage tracking within Databricks workspaces. The tutorial demonstrates setting up a metastore at the account level, creating catalogs and schemas, managing permissions through groups, and using quality monitoring dashboards for tables.
Delta Lake 3.3.0
Delta Lake 3.3.0 adds Identity Columns for automatic unique keys, VACUUM LITE for faster transaction log-based cleanup, and enables Row Tracking backfill on existing tables for row-level lineage tracking. UniForm Iceberg can now be enabled on existing Delta tables without data rewriting, and Type Widening is now supported in Delta Kernel for reading type-evolved tables.
EventsThe Best Data Warehouse is a Lakehouse
Databricks SQL provides data warehousing capabilities on a lakehouse architecture, integrating core database features, governance, and business intelligence integrations. The platform utilizes AI systems to deliver automated query optimization, performance scaling, and built-in vector search functions.
ReleasesMonitoring Databricks with System Tables
This video demonstrates how to use Databricks system tables within Unity Catalog to monitor workspace billing, data lineage, audit logs, and compute cluster history. It provides example SQL queries to track resource costs and analyze platform usage while explaining the current limitations compared to direct application logging.
NewsCross-Platform Data Lineage with OpenLineage
The presentation introduces OpenLineage as an open standard for collecting cross-platform data lineage metadata in real time to improve data observability and reduce troubleshooting friction. The speakers demonstrate how to configure and use OpenLineage with tools like Airflow, Spark, Flink, and dbt to track pipeline runs and visualize dependencies in Marquez.
NewsMulticloud Data Governance on the Databricks Lakehouse
Databricks presents multi-cloud data governance on the lakehouse using Unity Catalog to manage users, security, and data lineage across AWS, Azure, and Google Cloud. The session demonstrates how to implement this architecture using Delta sharing and Terraform to securely share data and synchronize entitlements across different cloud regions and business units.
NewsLineage System Table in Unity Catalog
The video introduces the lineage system table in Databricks Unity Catalog, which provides end-to-end data and AI lineage observability stored as standard Delta tables. It demonstrates how to use these system tables to troubleshoot data pipeline outages, analyze asset usage, and optimize costs by identifying wasted resources from unvisited jobs.
NewsActivate Your Lakehouse with Unity Catalog
Databricks demonstrates how Unity Catalog addresses lake house growing pains by simplifying data access, improving team collaboration, and optimizing storage and compute costs. The session also details a zero-downtime migration strategy from Hive metastore to Unity Catalog using custom Spark wrappers.
NewsAdvanced Governance with Collibra on Databricks
The video demonstrates how to integrate Collibra with Databricks Unity Catalog for advanced enterprise data governance. It showcases capabilities such as automated metadata ingestion, data profiling, lineage tracking, and the enforcement of row-level security and column-level masking policies.
CommunitySponsored by: Avanade | Enabling Real-Time Analytics with Structured Streaming and Delta Live Tables
The video demonstrates how Delta Live Tables simplifies data pipeline creation by automating infrastructure management, data quality checks, and performance optimization through familiar Python and SQL APIs. It also highlights the integration of Delta Live Tables with Unity Catalog to provide unified data governance, physical and logical data isolation, and comprehensive end-to-end lineage tracking.
NewsPII Detection at Scale on the Lakehouse
This video demonstrates an open source personally identifiable information detection module built on Databricks and Spark to scan both structured and unstructured storage. It covers custom named entity recognition, optical character recognition for sensitive documents, MLflow model wrapping, and Unity catalog integration for access control and lineage.
TutorialsLearn How to Reliably Monitor Your Data and Model Quality in the Lakehouse
Databricks announced lake house monitoring, a universal public preview solution designed to track data and AI asset quality directly within the unity catalog. The demonstration showed how the platform automatically generates metrics, custom dashboards, lineage-based root cause analysis, pii detection, and automated alerts for both data engineering and machine learning pipelines.
TutorialsA Technical Deep Dive into Unity Catalog's Practitioner Playbook
Databricks Unity Catalog provides centralized governance, lineage tracking, and security controls for data and AI assets across multi-cloud environments. The video demonstrates how to sync legacy data, configure role-based access, and implement query federation and data sharing using Unity Catalog.
NewsWhat’s New in Unity Catalog -- With Live Demos
Unity catalog provides a unified governance framework for managing secure data and AI assets across centralized, federated, and open ecosystems. The presentation demonstrates new features including column and row-level masking, volumes for non-tabular data, model management, lake house federation, automated lineage, and predictive optimization.
NewsConnecting the Dots with DataHub: Lakehouse and Beyond
DataHub is an open source metadata platform that provides a unified graph of a company's data ecosystem, including datasets, dashboards, and pipelines. The video demonstrates how DataHub achieves comprehensive metadata management through full-stack integrations, shifting metadata practices to source developer workflows, and enabling active operational controls like circuit breakers.
TutorialsOvalEdge: End-To-End Data Governance
OvalEdge provides a comprehensive, end-to-end data governance solution that functions as a catalog and integrates with numerous data sources and BI tools. The platform features automated data discovery, remote access management, lineage tracking, a business glossary, and data quality monitoring to streamline data governance across an organization.
TutorialsUnity Catalog: Journey to Unified Governance for Your Data and AI Assets on Lakehouse
The speakers present Unity Catalog as a solution for unified data and AI governance on the Lakehouse platform, highlighting its centralized metadata, access control, auditing, search, and automated lineage features. A live demonstration showcases creating storage credentials, external locations, managed and external tables, granting permissions, and exploring lineage.
NewsComplete Data Security and Governance Powered by Unity Catalog and Immuta
Databricks Unity Catalog and Immuta provide a unified approach to data security and governance. The video demonstrates how integrating Unity Catalog with Immuta enables automated data lineage, attribute-based access control, and dynamic policy enforcement.
TutorialsDiscover Data Lakehouse With End-to-End Lineage
Databricks data lineage automatically captures and visualizes relationships between data entities, jobs, notebooks, and dashboards. The demo illustrates how users can explore lineage graphs at both the table and column level within the data explorer.
NewsEnable Production ML with Databricks Feature Store
Databricks Feature Store helps data scientists and production engineers solve data challenges in production machine learning. The video demonstrates how to use the feature store to make machine learning models ready for production.
NewsComputational Data Governance at Scale
Fozzy Group, a holding of about 40 businesses, implements Data Mesh using federated computational data governance to tackle unclear data ownership. It covers building global data lineage across 200k tables, 40k table replications and 70k SQL stored procedures, plus lessons from automated tracking of ownership and data quality.
ReleasesSpline: Central Data-Lineage Tracking, Not Only For Spark
Spline, an open-source data lineage tool that began with Apache Spark, now offers a generic API and model that aggregates lineage metadata from many data tools into an end-to-end view of data flows. Its architecture, limitations, deployment options, comparison with other open-source tools and integration through the OpenLineage standard are covered.
NewsMachine Learning Data Lineage with MLflow and Delta Lake
MLflow Model Registry and Delta Lake Time Travel work together to give a full data lineage across machine learning life cycles. The registry traces each model version back to its original run and source code, Time Travel retrieves any historical data version, and a live demo combines both in an ML pipeline.
NewsBuilding a Federated Data Directory Platform for Public Health
Australia's National Health Services Directory, managed by Healthdirect Australia, uses a Federated Data Directory Platform that ingests authoritative sources and performs validation, matching, merging, enrichment and versioning with comprehensive lineage and provenance. The architecture runs on Databricks Delta Lake and Spark Structured Streaming, with source ranking from manual audits and planned ML classification.