Workflows

Databricks End-To-End Project | Zero-To-Expert | Streaming, AI, Lakeflow, Unity Catalog, AI/BI

This video demonstrates building an end-to-end restaurant analytics platform on Databricks, covering streaming and batch data ingestion, AI-powered sentiment analysis, and dashboard creation. It teaches how to use Unity Catalog, Lake Flow Connect for CDC, Spark declarative pipelines for real-time data from Event Hub, and how to construct a medallion architecture with fact and dimension tables.

Orchestration With Lakeflow Jobs

Lakeflow Jobs is a native orchestrator for Databricks that automates data workflows through scheduling, data-triggered execution, and built-in observability without requiring external tools. The presentation demonstrates building an end-to-end ETL pipeline with data ingestion from Salesforce, transformation via Delta Live Tables, and automated dashboard updates, all configured through a visual UI.

Databricks as the Backbone of MLOps: From Orchestration to Inference

From Days to Seconds — Reducing Query Times on Large Geospatial Datasets by 99%

The Global Water Security Center reduced geospatial query times from days to seconds by implementing Databricks medallion architecture with H3 spatial indexing and autoloader data ingestion on 30-billion-row datasets. This achieved a 99% query time reduction, saving approximately $1 million annually in labor costs and enabling analysts to self-serve analysis workflows through automated notebook orchestration.

Scaling Data Engineering Pipelines: Preparing Credit Card Transactions Data for Machine Learning

Delta Lake's partitioning, z-ordering, and file compaction reduced Mastercard's 10+ petabyte transaction table file count by 70% and improved query performance by 80%. Databricks Workflows' foreach operator enabled parallel processing across multiple clusters, completing nine months of graph network computation in two days.

Creating a Custom PySpark Stream Reader with PySpark 4.0

The video demonstrates how to create a custom PySpark 4.0 streaming data source by implementing Python classes that inherit from the new data source interfaces. It walks through building a custom stream reader to ingest data from an unsupported messaging system like Apache Active MQ directly into Delta tables.

Developer Best Practices on Databricks: Git, Tests, and Automated Deployment

Implement Git version control with branching and automated testing—unit, integration, and system tests—before merging code to maintain quality. Automate deployments across isolated environments using Databricks Asset Bundles and GitHub Actions to ensure consistent, error-free releases.

Five Things You Didn't Know You Could Do with Databricks Workflows

The video demonstrates advanced Databricks Workflows features including serverless jobs, repair and rerun capabilities for failed tasks, and late-job notifications. It also covers orchestrating external tools like DBT, managing modular jobs, and utilizing file arrival triggers for automated orchestration.

Embracing the Future of Data Engineering: The Serverless, Real-Time Lakehouse in Action

Databricks demonstrates a serverless real-time lakehouse architecture by streaming mobile phone sensor data into a cloud pipeline. The presentation features live ingestion through Kinesis, Delta Live Tables, Apache Spark structured streaming, and cross-cloud data sharing using Delta Sharing.

US Army Corp of Engineers Enhanced Commerce & National Sec Through Data-Driven Geospatial Insight

The video details how the U.S. Army Corps of Engineers modernizes its National Dredging Quality Management program by migrating from a legacy on-premises Oracle stack to a Databricks lake house architecture. It demonstrates how this cloud migration processes billions of streaming vessel telemetry points in under a minute to enhance real-time geospatial analytics, automation, and decision-making for national waterways.

What’s New in Databricks Workflows -- With Live Demos

Databricks workflows provides a unified lake house orchestrator that simplifies running complex tasks and managing dependencies. The platform features new capabilities such as conditional branching, visual monitoring, automated data triggers, and serverless compute options.

Databricks Asset Bundles: A Standard, Unified Approach to Deploying Data Products on Databricks

Databricks asset bundles allow teams to use YAML configuration files and the Databricks CLI to write code once and deploy data products consistently across development, staging, and production workspaces. The tool helps standardize CI/CD workflows, co-version project code with configurations, and manage cloud resources such as jobs and Delta live tables.

Introduction to Data Engineering on the Lakehouse

The video explains how the Databricks lake house platform unifies data lakes and data warehouses to simplify data engineering, governance, and analytics. It demonstrates building automated data pipelines using Delta live tables for ingestion and transformation, and databricks workflows for orchestration.

Orchestration Made Easy with Databricks Workflows

Databricks Workflows provides reliable orchestration for data, analytics, and AI on the lakehouse, enabling users to orchestrate tasks across any platform and any cloud. The video demonstrates how to use the Databricks Workflows user interface to easily create, monitor, and manage complex jobs and pipelines without writing code.

Migrate Your Existing DAGs to Databricks Workflows

Databricks Workflows enables users to migrate existing Directed Acyclic Graphs (DAGs) and consolidate orchestration within a single lakehouse environment. The video demonstrates how to recreate simple and complex Airflow DAGs using Databricks notebooks, SQL queries, Python scripts, and dbt tasks.

Turbocharge your AI/ML Databricks workflows with Precisely

Precisely integrates with Databricks to provide a location enabled lakehouse that enriches business data with precise IDs, geospatial attributes, and dynamic weather conditions. This platform allows data scientists and analysts to analyze risk, manage assets, and improve decision making using spatial datasets and machine learning workflows.