ETL
Recent items mentioning ETL across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Databricks is actively targeting the elimination of dedicated ETL pipelines through HTAP architectures 2 and built-in search in Lakebase Postgres, which runs hybrid search directly on operational data 4. For teams maintaining existing transformation stacks, Databricks outlined a cutover strategy to repoint dbt pipelines to the platform using Lakeflow Jobs for production orchestration 3.
Generated daily from the 4 most recent items mentioning ETL. Click any [N] to jump to the source.
What's the most common reason your ETL pipelines fail in production?
I've seen ETL pipelines fail for different reasons, from unexpected schema changes to missing data and transformation errors. I'm curious to know what causes the most issues in real-world projects. Is it usually source system changes, bad data, SQL errors, connection issues, or something else? Also, how do you usually identify the root cause and prevent the same issue from happening again? Would love to hear about your experiences and the problems you deal with most often. submitted by /u/ETL_Pipeline_Dev [link] [comments]
What is HTAP? A plain-English guide to hybrid transactional/analytical processing
Hybrid Transactional/Analytical Processing (HTAP) eliminates the latency and complexity of separate batch ETL pipelines by running analytical queries directly on fresh operational data. Evaluate the architectural tradeoffs between converged and composed designs, compare HTAP against Lake Transactional/Analytical Processing (LTAP), and learn key considerations for integrating these systems into a modern data stack.
How to Repoint dbt ETL Pipelines to Databricks
Existing dbt projects can be repointed from any source warehouse to Databricks with minimal code changes. Follow practical guidance covering adapter setup, namespace mapping, SQL dialect differences, production scheduling with Lakeflow Jobs, and model-by-model validation for a safe cutover.
Lakebase Search: State-of-the-art full text and vector search for Postgres
Lakebase Postgres now includes a built-in search engine, generally available on AWS and Azure, that enables native semantic, keyword, and hybrid search directly alongside operational data without separate ETL pipelines. Featuring a serverless architecture that scales to zero and decouples storage from compute, it delivers twice the throughput at a quarter of the cost of cloud Postgres with pgvector on 100-million-vector benchmarks.
Meet SDP Rewind: An undo button for your ETL pipelines
TutorialsHow to Connect Databricks and Fabric (No Copying Required)
Databricks catalog federation and Fabric mirroring enable zero-copy data access between platforms, allowing Fabric compute to read Databricks data or Databricks compute to read OneLake data without duplicating storage or managing ETL pipelines. The demonstration covers practical setup steps, required prerequisites like storage credentials and external data access permissions, and current limitations including read-only access and support for table objects only.
Inside the infrastructure strategies propelling AI leaders
Decoupling compute from low-cost lake storage through an AI-ready database allows teams to unify operational and analytical data without building costly ETL pipelines. This elastic architecture supports autonomous AI agents with instant environment provisioning and scale-to-zero compute, eliminating idle capacity costs while accelerating production cycles.
Postgres data stored in Parquet on S3: LTAP architecture explained
--- top comments --- [dsauerbrun] Maybe I'm too stupid to understand the article... How does this achieve performant querying for olap and oltp purposes? Based on my understanding, olap queries will go to the parquet files which are stored in a columnar fashion and oltp style queries will go to a caching layer that sits on top of those parquet files? What's the special sauce here? Seems like they're just caching the data which, for all intents and purposes, seems like the same solution of storing another copy of the data which is what they say they're avoiding. [ianberdin] Surprisingly, I'm already encountering a second solution that involves storing data chunks on S3 — and this is all within the same week. This is becoming popular. At Playcode, we built what we believe is a revolutionary file system for our Playcode Cloud (https://playcode.io/cloud), which enables the creation of full-stack web software. The FS built completely from scratch using Rust. We thought we were the smartest ones around and that nobody else had figured this out. But it turns out Databricks, Neon, and several others have as well. The idea behind a *Bottomless File System* is really cool, and it works very well for us. Essentially, as described here: - There is a *page server* - A *Linux file system* split into chunks (let's call them chunks instead of pages) - A *cache on NVMe* - And of course, *object storage*, where everything is asynchronously synchronized It works quite well, though it has its downsides. One clear advantage is that NVMe drives have become expensive lately, while object storage remains cheap — so the benefits are undeniable. That said, latency is also a factor. On top of that, uplink costs are rising. To run an object storage-backed file system, you need a very strong uplink with consistent speed — 1 Gbps is simply not enough. Ideally, you want *5 to 10 Gbps*, depending on the load. We spend a lot of time optimizing and experimenting with different hosting providers — specifically bare metal hardware. The main challenges are: - *Slow disks* - *Slow uplink* - And as it turns out, *object storage can be unreliable* — unless you're using S3 But AWS hardware is expensive, so nothing in life is ever that simple. [Avalaxy] Super cool stuff. Being able to combine your analytical platform and transactional database into one storage layer without having to set up ETL pipelines in between is really a game changer. Especially since it's just postgres, instead of some proprietary database. [saisrirampur] But why? I’m skeptical of the idea of unifying storage just because it sounds “elegant” or “cool”. It’s not obvious to me how a single storage engine can compete with purpose-built OLTP and OLAP systems like Postgres and ClickHouse, without significant tradeoffs. You also mention removing CDC pipelines. I’m curious if the materialization (conversion across formats) can catchup to an OLTP workload that is heavy (50K+ tps), which is pretty common these days. Also CDC if done right and with care can be magical for users and stays native to the OLTP/OLAP data-store. Third, data Lakes and open formats are suitable for Data Warehousing / Data analyst use-cases than real-time customer facing apps. Sure, you might work on changing that, which is what you are upto, but you’ll always run into tradeoffs, which will make it hard to unleash the best performance, much needed for the latter category. [andrenotgiant] Here's what I don't understand: Part of the value of doing an ETL pipeline via streaming replication is you get the full history of data in a table. An SCD type 2 table where each row also has a valid_from and valid_to timestamp column. How would someone do the same thing with this architecture?
Building a SQL ETL Pipeline: The Complete Guide for Data Engineers
Build a SQL ETL pipeline end-to-end, leveraging modern declarative SQL to empower SQL-native practitioners to own and operate data pipelines. Learn best practices for idempotency, modularization, governance, and automated testing to eliminate the production gap between analysts and data engineers.
TutorialsAI Agents That Remember: Building Stateful Systems with Lakebase
AI agents require four types of memory (working, episodic, entity, procedural) to be truly intelligent and stateful, which traditional databases struggle to provide. Databricks Lakebase, built on Postgres, offers a unified OLTP and OLAP solution with features like serverless auto-scaling and Git-style branching to manage these complex memory needs for AI agents.
Anyone have insights on pivoting from cloud engineering with Databricks administration or other regular IT into a Databricks data engineering role?
I've been in IT for the last 24ish years - started from the helpdesk, got experience and certs, fits and starts, etc. I've been doing Azure cloud engineering for the last 8ish years. In my previous job, I was asked to spin up an Azure Databricks test environment for our data science/data engineering teams. It grew, it got more mature, and by the end of my time there I was doing a lot of the administrative stuff - cluster policies, cost management, provisioning through SCIM, and the occasional technical question. I don't really have a background in databases or development; I've never written anything in Python or my own SQL queries but I've had plenty of situations where a dev/DBA would walk me through their code or query and show me what it did, after which point I'd break it down for troubleshooting. My current role with Microsoft has a subject matter expert team in Azure Databricks. I joined up with the team, had a lot of training on how the back end operates and how the data science/eng functionality works with Python and otherwise. I've been taking tickets with this SME team and done pretty well. I took the beta exam for the DP-750 Azure Databricks data engineering cert and just found out yesterday that I passed. Cloud engineering has become a lot less lucrative or Azure-focused as it was a few years ago and I've been exploring pivoting into different parts of IT. Apparently I know Databricks decently, but I know that's not nearly enough to find a data engineering role. Has anyone else been in this situation? How did you make your pivot? Did you take on projects in your current roles and spin them on your CV as data engineering work? Did you take your experience with DevOps pipelines and parlay it over to ETL pipelines? Any guidance or input would be much appreciated.
Backstage with Lakebase, part 2
Lakebase enables running production OLTP applications like Backstage on a serverless Postgres surface within Databricks, offering 1-second database branching and sub-4-second point-in-time recovery for schema migrations. Unity Catalog unifies governance for operational databases, providing single SQL query auditing, automatic row-level security propagation to branches, and zero-ETL cost attribution for FinOps.
How nOps Rebuilt Their Cloud Optimization Platform on Databricks Lakebase, and Why Other ISVs Should Too
nOps migrated its production cloud cost optimization platform to Databricks Lakebase, replacing a separate relational database and custom ETL pipelines with managed PostgreSQL directly integrated into the Lakehouse. The move provides an architectural blueprint for ISVs looking to eliminate synchronization overhead and unify operational and analytical data under a single governance model.
Data Science vs Data Engineering: Choosing Analysis or Infrastructure
BI reporting bridges raw data and operational teams by collecting, analyzing, and presenting data in structured formats. Effective BI relies on clean, integrated data flowing through ETL pipelines into a central repository, supporting both managed and ad hoc reporting.
Operational databases: How they work and when to use them
Databricks is introducing the "Lakebase," a new open architecture combining transactional database speed with data lake flexibility and economics, designed to overcome the limitations of traditional operational databases for modern unstructured data and AI workloads. This allows for real-time processing and concurrent transactions directly on the data lake, eliminating slow ETL pipelines and supporting diverse data types.
TutorialsYour Delta Tables Deserve a Postgres Home
Databricks demonstrates syncing Delta tables from Unity Catalog to a Postgres database within Lake Basin, enabling OLTP-style quick lookups for applications. Users can configure continuous, on-demand snapshot, or triggered sync modes, defining primary keys and grouping tables into pipelines for efficient data transfer.
NewsLakebase: Postgres That Actually Likes Your Lakehouse
Lakebase is a new Databricks offering that provides a fully managed, autoscaling PostgreSQL database designed to bridge the gap between analytical and transactional workloads in a lakehouse architecture. It features bidirectional data streaming between Delta tables and PostgreSQL, database branching for isolated development, and Unity Catalog governance.
NewsMaster Dimensional Modeling Lesson 03 - Understand the ETL Pipeline
The video explains the typical stages of a data warehouse ETL pipeline, including pre-staging (raw data), staging (cleaned data), operational data store (snapshot), and data mart (star schema). It also details the benefits of having multiple stages, such as easier debugging, data recovery, and auditability, and how this maps to the Medallion Architecture (Bronze, Silver, Gold).
Tutorials52 Lakeflow Spark Declarative Pipelines | New Pipeline Code Editor | AUTO CDC |External Target Sinks
Databricks' LakeFlow Spark Declarative Pipelines (SDP), formerly Delta Live Tables (DLT), offers a unified solution for data ingestion, transformation, and orchestration, now open-sourced with Apache Spark 4.1. The video demonstrates using the new pipeline code editor to build SDPs in Python and SQL, showcasing features like auto CDC (formerly apply changes) and external target sinks.
Events[Demo] Lakeflow Designer: No-Code ETL, Powered by the Data Intelligence Platform
Lakeflow Designer allows users to create ETL pipelines using a no-code approach. It features a "transform by example" assistant that can generate data transformations from a screenshot of desired output.
EventsWhat Should You Do With Lakebase — Explained by Databricks Co-founder Reynold Xin
Lakebase offers an enterprise-ready relational database solution for new applications, serving existing data like ML feature stores, and simplifying complex ETL pipelines. It integrates with Databricks infrastructure, providing features like security, compliance, and governance.
TutorialsOrchestration With Lakeflow Jobs
Lakeflow Jobs is a native orchestrator for Databricks that automates data workflows through scheduling, data-triggered execution, and built-in observability without requiring external tools. The presentation demonstrates building an end-to-end ETL pipeline with data ingestion from Salesforce, transformation via Delta Live Tables, and automated dashboard updates, all configured through a visual UI.
TutorialsHealthcare Interoperability: End-to-End Streaming FHIR Pipelines With Databricks & Redox
Redox streams healthcare data from EHR systems into Databricks by converting various formats (HL7, XML, X12) to FHIR JSON with normalization and connectivity handling. Databricks processes FHIR data using SQL and AI without specialized software, enabling real-time analytics and bidirectional data exchange for healthcare use cases like prior authorization.
TutorialsFrom Apache Airflow to Lakeflow Jobs: A Guide for Workflow Modernization
Lakeflow provides unified data ingestion, transformation, and orchestration capabilities within Databricks that can be orchestrated through Apache Airflow using operators like SQL queries, materialized views, and AI sentiment analysis functions. Databricks supports both Lakeflow jobs as a fully managed integrated platform and Apache Airflow as a self-hosted customizable option, giving users choice based on their specific requirements.
TutorialsRace to Real-Time: Low-Latency Streaming ETL With Next-Gen OLTP-DB
The video demonstrates building a low-latency streaming ETL solution using Databricks and Lakebase, achieving sub-two-second end-to-end latency for operational workloads. It teaches key performance optimization techniques, including controlling microbatch sizes, leveraging broadcast joins, enabling dynamic file pruning, and utilizing liquid clustering to minimize data movement and object storage costs.
TutorialsGetting the Most Out of DLT: A Deep Dive on What’s New and Best Practices
Databricks' Declarative Pipelines (rebranded from Delta Live Tables) allow users to write production-grade ETL pipelines in just a few lines of SQL or Python by abstracting away checkpoint management, streaming logic, and failure handling. The system uses an optimizer called Enzyme to automatically select the most efficient incremental computation strategy—whether materialized views, streaming tables, or change data capture patterns—based on query type and data characteristics.
EventsData + AI Summit Keynote Day 2
Databricks announced major expansions to its data platform including managed Apache Iceberg tables in Unity Catalog with governance across multiple engines, and LakeFlow Designer for building production ETL pipelines without code. The company also contributed real-time streaming mode and declarative pipelines to Apache Spark, and launched Zerobus, a direct write API enabling terabytes-per-hour data ingestion into the lakehouse.
NewsMaster Data Workload Automation: Introduction
This video teaches the differences between job schedulers, standard orchestrators, and data orchestrators for automating data workloads like ETL pipelines. It demonstrates how to choose the right tool based on system complexity, highlighting simple schedulers like cron and SQL Server agent alongside advanced data orchestrators like Dagster and Airflow.
Tutorials122. Databricks | Pyspark| Delta Live Table: Introduction
This video introduces Databricks Delta Live Tables as a declarative development framework designed to simplify and accelerate ETL pipeline creation. It details traditional pipeline challenges like complex medallion architectures, manual data quality checks, and difficult monitoring, while demonstrating how Delta Live Tables automate infrastructure management and error handling.
ReleasesNebula: The Journey of Scaling Instacart’s Data Pipelines with Apache Spark™ and Lakehouse
Instacart migrated its large-scale advertising and data pipelines from a traditional data warehouse to a Databricks lake house powered by Apache Spark and Delta Lake. The presenters demonstrate how this transition unified batch and streaming data processing, reduced storage and compute costs for over 40 petabytes of data, and improved developer productivity through Scala-based modular pipelines and local unit testing.
NewsIncreasing Data Trust: Enabling Data Governance on Databricks Using Unity Catalog & ML-Driven MDM
The speakers explain how Effective integrated data governance with a Databricks lake house and Unity catalog to build scalable master data management. They demonstrate how combining machine learning, natural language processing, and graph frames with business domain expertise reduces duplicate customer records.
NewsDetermining When to Use GPU for Your ETL Pipelines at Scale
The video teaches how to use the open-source Spark Rapids qualification tool to evaluate Apache Spark event logs and identify which ETL pipeline jobs are suitable for GPU acceleration. It demonstrates an automated workflow developed by AT&T that analyzes job logs at scale, calculates projected speedups and cost savings, and prioritizes conversions based on job execution frequency.
NewsVector Data Lakes
Vector databases function like online transactional stores and become too expensive for storing large historical embeddings in generative AI applications. Moving vector data to scalable data lake formats like Parquet and Lance enables cost-effective batch querying and seamless integration with distributed compute engines.
NewsIntroduction to Data Engineering on the Lakehouse
The video explains how the Databricks lake house platform unifies data lakes and data warehouses to simplify data engineering, governance, and analytics. It demonstrates building automated data pipelines using Delta live tables for ingestion and transformation, and databricks workflows for orchestration.
NewsLearn to Efficiently Test ETL Pipelines
This video teaches how to efficiently test ETL pipelines by using Saff Squeeze techniques to isolate and diagnose bugs with bug-specific tests. It demonstrates how to apply multi-layered testing strategies to reduce data pipeline debugging time and prevent regressions.
NewsLog Processing at Scale
Log Processing at Scale explains how Pure Storage uses Spark and Databricks to process and analyze large volumes of test and platform logs efficiently. The presentation demonstrates an automated ELT pipeline called Autotriage that scans log streams for known failure signatures to accelerate debugging and reduce engineering effort.
NewsPresto On Spark: A Unified SQL Experience
Presto on Spark is a library that allows Presto queries to run on a Spark cluster, offering a unified SQL experience by combining Presto's ANSI SQL dialect and low latency with Spark's scalability and fault tolerance. Intuit's SuperGlue data platform successfully implemented Presto on Spark to solve challenges related to dialect conversion, query performance, and cluster scaling across thousands of daily jobs.
NewsHow Databricks Leverages Auto Loader to Ingest Millions of Files an Hour
Auto Loader, a Structured Streaming source in Databricks, handles continuous, incremental ingestion from cloud storage with efficient file discovery through notifications or incremental listing and exactly-once guarantees across billions of files. The talk covers schema inference, drift detection and evolution, and how Databricks uses Auto Loader internally to ingest millions of files every hour.
NewsEmpowering Real Time Patient Care Through Spark Streaming
Takeda's Plasma Derived Therapies unit uses Spark Streaming on Databricks to track patient interactions at plasma donation centers in real time, replacing costly Lambda, step function and triggered-job ETL. Streams merge CSV files landed in S3 by AWS DMS and Informatica into Delta tables, enrich records into RDS and feed gold analytic streams.
NewsSpeed up UDFs with GPUs using the RAPIDS Accelerator
User-defined functions appear as opaque transforms that can stop the RAPIDS Accelerator for Apache Spark from running query operations on the GPU. The RAPIDS Accelerator UDF Compiler translates some simple UDFs into GPU-processed Catalyst operations, and users can write GPU versions of Scala, Java or Hive UDFs for maximum control.
News10 Things Learned Releasing Databricks Enterprise Wide
Western Governors University shares 10 lessons from rewriting its ETL pipelines in Scala and Python and moving its Enterprise Data Warehouse into Delta on Databricks, growing from 4 to over 120 users across 8 business units. Topics include AWS and Databricks user management, cost control, custom code management pipelines and money-saving Apache Spark snippets.
CommunityEmpowering Zillow’s Developers with Self-Service ETL
Zillow engineers present two internal self-service platforms that let data analysts and data producers build, maintain and monitor ETL pipelines without handling orchestration, deployment or Apache Spark processing. They explain why each user group got its own interface, how much abstraction each received, and how the Spark package Pipeler supports reliable pipelines.
NewsOn Improving Broadcast Joins in Apache Spark SQL
Workday raises the relation size threshold under which Spark SQL broadcast joins stay practical, using executor-side broadcast that sends data directly between executors instead of through the driver. Whole-stage code generation changes support the higher threshold while broadcast join memory in executors stays limited, and findings from large production ETL jobs are shared.
NewsBuilding the Petcare Data Platform using Delta Lake and 'Kyte': Our Spark ETL Pipeline
Mars Petcare's Kinship Data & Analytics division builds its Azure-based Petcare Data Platform on Delta Lake, replacing Azure Data Factory with Kyte, a bespoke Spark and Databricks pipeline tool for ingesting, cleansing and processing data. Delta Lake also stores ETL configuration, and a custom UI monitors and schedules the Spark pipelines.
NewsUsing Apache Spark for Predicting Degrading and Failing Parts in Aviation
Naval aviation teams use Spark for ETL across large, heterogeneous aviation data lakes and for component health reporting, with Spark ML regression models flagging anomalous flights for human review. Parallelized health scoring cuts model runtimes from days or weeks to hours and has flagged parts before failure, though visualization still happens in Pandas.
NewsData Science Across Data Sources with Apache Arrow
Apache Arrow, an open source columnar in-memory data representation, lets analytical systems exchange and process data across sources without copying it into one location. The talk outlines how open source projects use Arrow for performance and interoperability, including a 50x speedup in PySpark for Spark-Pandas interoperability, and analysis across disparate data sources.
NewsModern ETL Pipelines with Change Data Capture Thiago Rigo GetYourGuide - David Mariassy
GetYourGuide rebuilt its ETL pipeline from scratch with Debezium, Kafka, Spark and Airflow, using Debezium for change data capture from database binlogs streamed to Kafka and handling schema changes automatically. Built with Databricks to reduce operation time, the pipeline replaces an error-prone daily legacy system and refreshes the data lake multiple times a day.